Upgrade android_hardware_smoke_test CI to run instrumented tests (#189390)

This PR adds native Android instrumented tests to the
`android_hardware_smoke_test` CI suite in parallel to existing driver
tests (Part of https://github.com/flutter/flutter/issues/187538).

Adds two new shards temporarily, both with `bringup: true`:
* `android_hardware_smoke_vulkan_instrumented_tests` 
* `android_hardware_smoke_opengles_instrumented_tests`

These shards run the driver tests first, similar to existing shards,
then subsequently run the instrumented tests. They do this via a new
optional `runInstrumented` param in
`run_android_hardware_smoke_tests.dart`.

We'll remove the parameter and put this behavior in the existing shards
after it proves stable.

## Why do we need to run the instrumented tests sequentially after the
driver tests?

I generally prefer to avoid doing multiple things sequentially in an
integration test. It tends to introduce flakiness. So for this suite,
why can't we have one shard for driver tests and a separate one for
instrumented tests?

The driver tests use a skia gold comparator to compare result images to
skia gold at runtime. However, no such comparator is available for
instrumented tests. `goldctl` isn't designed to run on android. So we
need to get the goldens somehow at APK build time.

In this PR, the integration tests run in two sequential passes. We first
execute a host-driven driver test to save reference screenshots, and
then run native JUnit tests to compare against those local files. This
is the same way we will generate instrumented test APKs to be delivered
to OEMs. I considered optimizing this by downloading the latest approved
goldens directly from the Skia Gold server. It turns out it's possible
to calculate the public URLs based on the test name and some other
parameters. So we could theoretically write a script which cheaply
fetches all the goldens for the suite. Then we could configure a shard
which runs that script instead of the driver tests, achieving an
independent shard for instrumented tests only.

However, this downloader approach is incompatible with presubmit. Any PR
that introduces a change to the goldens would fail because the script
would only return the latest approved goldens. Even after a developer
approved the change on the dashboard, subsequent presubmit runs would
still download the old goldens, deadlocking the PR checks. Technically
that's not a problem for post-submit runs, but I'd rather keep presubmit
and postsubmit identical instead of optimizing only postsubmit with
something like this.

Running the driver pass first, as we do in this PR, avoids the problem
by dynamically capturing reference screenshots generated by the PR's own
code. This keeps the skew at zero and allows tests to pass once
approved.

Also, I'm not that comfortable relying on calculating URLs, since there
would be no guarantee that the URL scheme will remain stable. The URL
scheme hasn't changed since 2020, but compared to relying only on
goldctl, it just feels more fragile to me. That being said, I've
implemented the download script in a personal agent skill instead, where
I can use it at my own risk, and it isn't weight bearing in any way.

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant documentation (doc comments with `///`).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
5 files changed
tree: 6c64bb17af190b76197b054964eb79455c7ef4a1
  1. .agents/
  2. .claude/
  3. .gemini/
  4. .github/
  5. .vscode/
  6. agent-artifacts/
  7. bin/
  8. buildtools/
  9. dev/
  10. docs/
  11. engine/
  12. examples/
  13. packages/
  14. third_party/
  15. .autoroller-preupload.sh
  16. .ci.yaml
  17. .direnv
  18. .gitattributes
  19. .gitignore
  20. analysis_options.yaml
  21. AUTHORS
  22. CHANGELOG.md
  23. CODE_OF_CONDUCT.md
  24. CODEOWNERS
  25. CONTRIBUTING.md
  26. dartdoc_options.yaml
  27. DEPS
  28. flutter_console.bat
  29. LICENSE
  30. PATENT_GRANT
  31. pubspec.lock
  32. pubspec.yaml
  33. README.md
  34. TESTOWNERS
README.md

Flutter CI Status Discord badge Twitter handle BlueSky badge codecov LFX Health Score CII Best Practices SLSA 1

Flutter is Google's SDK for crafting beautiful, fast user experiences for mobile, web, and desktop from a single codebase. Flutter works with existing code, is used by developers and organizations around the world, and is free and open source.

Documentation

For release and other announcements, join the flutter-announce mailing list. Our documentation also tracks breaking changes across releases.

Terms of service

The Flutter tool may occasionally download resources from Google servers. By downloading or using the Flutter SDK, you agree to the Google Terms of Service: https://policies.google.com/terms

For example, when installed from GitHub (as opposed to from a prepackaged archive), the Flutter tool will download the Dart SDK from Google servers immediately when first run, as it is used to execute the flutter tool itself. This will also occur when Flutter is upgraded (e.g. by running the flutter upgrade command).

About Flutter

We think Flutter will help you create beautiful, fast apps, with a productive, extensible and open development model, whether you're targeting iOS or Android, web, Windows, macOS, Linux or embedding it as the UI toolkit for a platform of your choice.

Beautiful user experiences

We want to enable designers to deliver their full creative vision without being forced to water it down due to limitations of the underlying framework. Flutter‘s layered architecture gives you control over every pixel on the screen and its powerful compositing capabilities let you overlay and animate graphics, video, text, and controls without limitation. Flutter includes a full set of widgets that deliver pixel-perfect experiences whether you’re building for iOS (Cupertino) or other platforms (Material), along with support for customizing or creating entirely new visual components.

Fast results

Flutter is fast. It's powered by hardware-accelerated 2D graphics libraries like Skia (which underpins Chrome and Android) and Impeller. We architected Flutter to support glitch-free, jank-free graphics at the native speed of your device.

Flutter code is powered by the world-class Dart programming language, which enables compilation to 32-bit and 64-bit ARM machine code for iOS and Android, JavaScript and WebAssembly for the web, as well as Intel x64 and ARM for desktop devices.

Productive development

Flutter offers stateful hot reload, allowing you to make changes to your code and see the results instantly without restarting your app or losing its state.

Hot reload animation

Extensible and open model

Flutter works with any development tool (or none at all), and also includes editor plug-ins for both Visual Studio Code and IntelliJ / Android Studio. Flutter provides tens of thousands of packages to speed your development, regardless of your target platform. And accessing other native code is easy, with support for both FFI (on Android, on iOS, on macOS, and on Windows) as well as platform-specific APIs.

Flutter is a fully open-source project, and we welcome contributions. Information on how to get started can be found in our contributor guide.