testing/scenario_app/android/README.md - mirrors/engine - Git at Google

 # Scenario App: Android Tests and Test Runner

 End-to-end tests and test infrastructure for the Flutter engine on Android.

 > [!IMPORTANT]
 > There are several **known issues** with this test suite:
 >
 > - [#144407](https://github.com/flutter/flutter/issues/144407): "Newer Android"
 >   tests are missing expected cropping and rotation.
 > - [#144365](https://github.com/flutter/flutter/issues/144365): Tests that use
 >   `ExternalTextureFlutterActivity` sometimes do not render.
 > - [#144232](https://github.com/flutter/flutter/issues/144232): Impeller Vulkan
 >   sometimes hangs on emulators.
 > - [#144352](https://github.com/flutter/flutter/issues/144352): Skia Gold
 >   sometimes is missing expected diffs.

 ---

 Top topics covered in this document include (but are not limited to):

 - [Running the Tests](#running-the-tests)
 - [Contributing](#contributing)
 - [Project History](#project-history)
 - [Troubleshooting](#troubleshooting)
 - [Getting Help](#getting-help)

 ## Introduction

 This package simulates a Flutter app that uses the engine (`dart:ui`) only, in
 conjunction with Android-specific embedding code that simulates the use of the
 engine in a real app (such as plugins and platform views).

 A custom test runner, [`run_android_tests.dart`](bin/run_android_tests.dart), is
 used to run the tests on a connected device or emulator, and report the results,
 including screenshots (golden files) and logs from `adb logcat`.

 In the following architecture diagram:

 > <a
 >   href="https://github.com/flutter/flutter/wiki/The-Engine-architecture"
 >   title="The Engine architecture"> <img
 >   width="300"
 >   alt="Anatomy of a Flutter app"
 >   src="https://raw.githubusercontent.com/flutter/engine/main/docs/app_anatomy.svg"
 > /> </a>

 - The Dart app is represented by [`lib/main.dart`](../lib/main.dart).
 - There is no framework code.
 - `dart:ui` and the engine are the same as in a real app.
 - Android-specific application code is in this directory
   ([`android/`](./)).
 - The runner is a _custom_ test runner,
   [`run_android_tests.dart`](bin/run_android_tests.dart).

 ### Scope of Testing

 The tests in this package are end-to-end tests that _specifically_ exercise the
 engine and Android-specific embedding code. They are not [unit tests][1] for the
 engine or the framework, and are not designed to test the behavior of the
 framework or specific plugins, but rather to simulate the use of a framework
 or plugins downstream of the engine.

 In other words, we test "does the engine work on Android?" without a dependency
 on either the Flutter framework, Flutter tooling, or any specific plugins.

 [1]: ../../../shell/platform/android

 ### Golden Comparisons

 Many of the Android-specific interactions with the engine are visual, such as
 [external textures](https://api.flutter.dev/flutter/widgets/Texture-class.html)
 or [platform views](https://docs.flutter.dev/platform-integration/android/platform-views),
 and as such, the tests in this package use golden screenshot file comparisons to
 verify the correctness of the engine's output.

 For example, in [`ExternalTextureTests_testMediaSurface`](https://flutter-engine-gold.skia.org/search?corpus=flutter-engine&include_ignored=false&left_filter=name%3DExternalTextureTests_testMediaSurface&max_rgba=0&min_rgba=0&negative=true&not_at_head=false&positive=true&reference_image_required=false&right_filter=&sort=descending&untriaged=true), a [video](app/src/main/assets/sample.mp4) is converted to an external texture and displayed in a Flutter app. The test takes a screenshot of the app and compares it to a golden file:

 <img
   alt="Two pictures, the top one Flutter and the bottom Android"
   src="https://github.com/flutter/flutter/assets/168174/e2c34b88-d03d-4732-87e4-a86c97d006c5"
   width="300"
 />

 _The top picture is the Flutter app, and the bottom picture is just Android._

 See also:

 - [`tools/compare_goldens`](../../../tools/compare_goldens).

 ## Prerequisites

 If you've never worked in the `flutter/engine` repository before, you will
 need to setup a development environment that is quite different from a typical
 Flutter app or even working on the Flutter framework. It will take roughly
 30 minutes to an hour to setup an environment, depending on your familiarity
 with the tools and the speed of your internet connection.

 See also:

 - [Setting up the Engine](https://github.com/flutter/flutter/wiki/Setting-up-the-Engine-development-environment)
 - [Rebuilding the Tests](#rebuilding-the-tests)

 ### Android SDK

 It's highly recommended to use the engine's vendored Android SDK, which once
 you have the engine set up, you can find at
 `$ENGINE/src/flutter/third_party/android_tools/sdk`. Testing or running with
 other versions of the SDK may work, but it's _not guaranteed_, and might have
 different results.

 Consider also placing this directory in the `ANDROID_HOME` environment variable:

 ```sh
 export ANDROID_HOME=$ENGINE/src/flutter/third_party/android_tools/sdk
 ```

 ### Device or Emulator

 The tests in this package require a connected device or emulator to run. The
 device or emulator should be running the same version of Android as the CI
 configuration (there are issues with crashes and other problems on older
 emulators in particular).

 > [!CAUTION]
 >
 > [#144561](https://github.com/flutter/flutter/issues/144561): The emulator
 > vendored in the engine checkout is old and [has a known issue with Vulkan](https://github.com/flutter/flutter/issues/144232).
 >
 > If you're working locally, you can update your copy by running:
 >
 > ```sh
 > $ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager --install emulator
 > ```

 ### Additional Dependencies

 While not required, it is **strongly recommended** to use an IDE such as
 [Android Studio](https://developer.android.com/studio) when contributing to the
 Android side of the test suite, as it will provide a better development
 experience such as code completion, debugging, and more:

 <img
   width="500"
   alt="Screenshot of Android Studio with Autocompletion"
   src="https://github.com/flutter/flutter/assets/168174/79b685f5-8da7-4396-abe6-9baeb29d7ce3"
 />

 > [!TIP]
 >
 > Android Studio is expected to work _out of the box_ in this directory.
 >
 > If you encounter any issues, please [file a bug](#getting-help).

 ## Running the Tests

 The [test runner](../bin/run_android_tests.dart) is a Dart script that installs
 the test app and test suite on a connected device or emulator, runs the tests,
 and reports the results including screenshots (golden files) and logs from
 `adb logcat`.

 From the `$ENGINE/src/flutter` directory, run:

 ```sh
 dart ./testing/scenario_app/bin/run_android_tests.dart
 ```

 By default when run locally, the test runner:

 - Uses the engine-wide default graphics backend for Android.
 - Uses the last built Android-specific engine artifacts (i.e. `$ENGINE/src/out/android_*/`).
 - Will not diff screenshots, but does save them to a logs directory.

 ### Rebuilding the Tests

 If you've made changes to any file in `scenario_app`, incluing the Dart code
 in `lib/` or the Android code in `android/`, you will need to rebuild the
 tests before running them.

 To rebuild the `scenario_app` for the `android_debug_unopt_arm64` variant:

 ```sh
 ninja -C out/android_debug_unopt_arm64 scenario_app
 ```

 See also:

 - [Compiling the Engine](https://github.com/flutter/flutter/wiki/Compiling-the-engine)

 ### Common Options

 A list of options can be found by running:

 ```sh
 dart ./testing/scenario_app/bin/run_android_tests.dart --help
 ```

 Frequently used options include:

 - `--out-dir`: Which engine artifacts to use (e.g.
   `--out-dir=../out/android_debug_unopt_arm64`).

 - `--logs-dir`: Where to save full `adb logcat` logs and screenshots.

 - `--[no]-enable-impeller`: Enables/disables use of the Impeller graphics
   backend.

 - `--impeller-backend`: Use a specific Impeller backend (e.g.
   `--impeller-backend=opengles`).

 - `--force-surface-producer-surface-texture`: Force the use of `SurfaceTexture`s
   for plugin code that uses `SurfaceProducer`s. This instruments the same code
   path that is used for Android API versions that are <= 28 without requiring
   an older emulator or device.

 - `--smoke-test=<full.class.Name>`: Runs a specific test, instead of all tests.

 ### Advanced Options

 When debugging the runner itself, you can use the `--verbose` flag to see more
 detailed output, including additional options such as configuring which binary
 of `adb` to use:

 ```sh
 dart ./testing/scenario_app/bin/run_android_tests.dart --help --verbose
 ```

 See also:

 - [`bin/utils/options.dart`](../bin/utils/options.dart).

 ### CI Configuration

 See [`ci/builders`](../../../ci/builders) and grep for `run_android_tests.dart`:

 ```sh
 grep -r run_android_tests.dart ci/builders
 ```

 > [!NOTE]
 > The Impeller OpenGLES backend tests are only run on staging (`bringup: true`)
 > and as such are **non-blocking**. We expect to stabilize and run these tests
 > as part of a wider release of the Impeller OpenGLES backend.

 #### Older Android

 "Older Android" refers to "code paths to support older Android API levels".
 Specifically, these configurations use
 `--force-surface-producer-surface-texture` (see [above](#common-options) for
 details).

 | Backend           | CI Configuration            | CI History                                     | Skia Gold                 |
 | ----------------- | --------------------------- | ---------------------------------------------- | ------------------------- |
 | Skia              | [`ci/builders`][skia-st-ci] | [Presubmit][skia-try], [Postsubmit][skia-prod] | [Skia Gold][skia-st-gold] |
 | Impeller OpenGLES | [`ci/builders`][imp-st-ci]  | [Staging][imp-staging]                         | N/A                       |

 [skia-try]: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_android_emulator_skia_tests
 [skia-prod]: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20linux_android_emulator_skia_tests
 [imp-staging]: https://ci.chromium.org/ui/p/flutter/builders/staging/Linux%20linux_android_emulator_opengles_tests
 [impeller-try]: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_android_emulator_tests
 [impeller-prod]: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20linux_android_emulator_tests
 [skia-st-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--no-enable-impeller%22+%22run_android_tests.dart%22+%22--force-surface-producer-surface-texture%22&type=code
 [skia-st-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dtrue%26GraphicsBackend%3Dskia&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia
 [imp-st-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dopengles%22+%22run_android_tests.dart%22+%22--force-surface-producer-surface-texture%22&type=code
 [imp-st-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dtrue%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia

 #### Newer Android

 | Backend           | CI Configuration             | CI History                                             | Skia Gold                |
 | ----------------- | ---------------------------- | ------------------------------------------------------ | ------------------------ |
 | Skia              | [`ci/builders`][skia-ci]     | [Presubmit][skia-try], [Postsubmit][skia-prod]         | [Skia Gold][skia-gold]   |
 | Impeller OpenGLES | [`ci/builders`][imp-gles-ci] | [Staging][imp-staging]                                 | N/A                      |
 | Impeller Vulkan   | [`ci/builders`][imp-vk-ci]   | [Presubmit][impeller-try], [Postsubmit][impeller-prod] | [Skia Gold][imp-vk-gold] |

 [skia-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--no-enable-impeller%22+%22run_android_tests.dart%22&type=code
 [skia-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dfalse%26GraphicsBackend%3Dskia&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia
 [imp-gles-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dopengles%22+%22run_android_tests.dart%22&type=code
 [imp-vk-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dvulkan%22+%22run_android_tests.dart%22&type=code
 [imp-vk-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dfalse%26GraphicsBackend%3Dimpeller-vulkan&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia

 ## Contributing

 [![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/flutter/flutter/e%3A%20scenario-app)](https://github.com/flutter/flutter/issues?q=is%3Aopen+is%3Aissue+label%3A%22e%3A+scenario-app%22)

 Contributions to this package are welcome, as it is a critical part of the
 engine's test suite.

 ### Anatomy of a Test

 A "test" in practice is a combination of roughly 3 components:

 1. An Android _[JUnit test][]_ that configures and launches an Android activity.
 2. An Android _[activity][]_, which simulates the Android side of a plugin or
    platform view.
 3. A Dart _[scenario][]_, which simulates the Flutter side of an application.

 [junit test]: ./app/src/androidTest/java/dev/flutter/scenariosui/DrawSolidBlueScreenTest.java
 [activity]: ./app/src/main/java/dev/flutter/scenarios/ExternalTextureFlutterActivity.java
 [scenario]: ../lib/src/solid_blue.dart

 While every test suite has exactly one JUnit-instrumented class, each test can
 have many activities and scenarios, each with their own configuration, setup,
 and assertions. Not all of this is desirable, but it is the current state of
 the test suite.

 A test might also take a screenshot. See the _Skia Gold_ links in
 [CI Configuration](#ci-configuration) for examples.

 ### Project History

 This test suite was [originally written in 2019](https://github.com/flutter/engine/pull/10007)
 with a goal of:

 > \[being\] suitable for embedders to do integration testing with - it has no
 > dependencies on the flutter_tools or framework, and so will not fail/flake
 > based on variances in those downstream.

 Unfortunately, the Android side of the test suite was never fully operational,
 and the tests, even if failing, were accidentally be reported as passing on CI.
 In 2024, as the team got closer to shipping our new graphics backend,
 [Impeller](https://docs.flutter.dev/perf/impeller) on Android, it was clear that
 we needed a reliable test suite for the engine on Android, particularly for
 visual tests around external textures and platform views.

 So, this package was revived and updated to be a (more) reliable test suite for
 the engine on Android. It's by no means complete
 ([contributions welcome](#contributing)), but it did successfully catch at least
 one bug that would not have been detected automatically otherwise.

 _Go forth and test the engine on Android!_

 ## Troubleshooting

 If you encounter any issues, please [file a bug](#getting-help).

 ### My test is failing on CI

 If a test is failing on CI, it's likely that the test is failing locally as
 well. Try the steps in [running the Tests](#running-the-tests) to reproduce the
 failure locally, and then debug the failure as you would any other test. If this
 is your first time working on the engine, you may need to setup a development
 environment first (see [prerequisites](#prerequisites)).

 The test runner makes extensive use of logging and screenshots to help debug
 failures. If you're not sure where to start, try looking at the logs and
 screenshots to see if they provide any clues (you'll need them to file a bug
 anyway).

 `test: Android Scenario App Integration Tests (Impeller/Vulkan)`
 on [LUCI](https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Engine%20Drone/2079486/overview) produces these logs:

 ![Screenshot of Logs](https://github.com/flutter/flutter/assets/168174/fed8fae6-bcd2-4e6e-a074-c2c37f34a770)

 The files include a screenshot of each test, and a _full_ logcat log from the
 device (you might find it easier trying the `stdout` of the test first, which
 uses rudimentary log filtering). In the case of multiple runs, the logs are
 prefixed with the test configuration and run attempt.

 ### Gradle is failing due to an "not part of the dependency lock state" error

 If you update dependencies in the `app/build.gradle` file, you may encounter an
 error (probably in CI) that looks like:

 ```txt
 FAILED: scenario_app/reports/lint-results.xml
 vpython3 ../../flutter/testing/rules/run_gradle.py /b/s/w/ir/cache/builder/src/flutter/testing/scenario_app/android lint --no-daemon -Pflutter_jar=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/flutter.jar -Pout_dir=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app --project-cache-dir=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app/.gradle --gradle-user-home=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app/.gradle

 FAILURE: Build failed with an exception.

 * What went wrong:
 Execution failed for task ':app:checkDebugAarMetadata'.
 > Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
    > Resolved 'org.hamcrest:hamcrest-core:1.3' which is not part of the dependency lock state
    > Resolved 'com.google.code.findbugs:jsr305:2.0.2' which is not part of the dependency lock state
 ```

 This is because [`gradle.lockfile`](./app/gradle.lockfile) is out of date. To
 update it, run:

 ```sh
 cd testing/scenario_app/android
 $ENGINE_SRC/flutter/third_party/gradle/bin/gradle app:dependencies --write-locks
 ```

 ## Getting Help

 To suggest changes, or highlight problems, please [file an issue](https://github.com/flutter/flutter/issues/new?labels=e:%20scenario-app,engine,platform-android,fyi-android,team-engine).

 If you're not sure where to start, or need help debugging or contributing, you
 can also reach out to [`hackers-android`](https://discord.com/channels/608014603317936148/846507907876257822) on Discord, or the Flutter engine team internally. There is
 no full-time maintainer of this package, so be prepared to do some of the
 legwork yourself.
	# Scenario App: Android Tests and Test Runner

	End-to-end tests and test infrastructure for the Flutter engine on Android.

	> [!IMPORTANT]
	> There are several known issues with this test suite:
	>
	> - [#144407](https://github.com/flutter/flutter/issues/144407): "Newer Android"
	> tests are missing expected cropping and rotation.
	> - [#144365](https://github.com/flutter/flutter/issues/144365): Tests that use
	> `ExternalTextureFlutterActivity` sometimes do not render.
	> - [#144232](https://github.com/flutter/flutter/issues/144232): Impeller Vulkan
	> sometimes hangs on emulators.
	> - [#144352](https://github.com/flutter/flutter/issues/144352): Skia Gold
	> sometimes is missing expected diffs.

	---

	Top topics covered in this document include (but are not limited to):

	- [Running the Tests](#running-the-tests)
	- [Contributing](#contributing)
	- [Project History](#project-history)
	- [Troubleshooting](#troubleshooting)
	- [Getting Help](#getting-help)

	## Introduction

	This package simulates a Flutter app that uses the engine (`dart:ui`) only, in
	conjunction with Android-specific embedding code that simulates the use of the
	engine in a real app (such as plugins and platform views).

	A custom test runner, [`run_android_tests.dart`](bin/run_android_tests.dart), is
	used to run the tests on a connected device or emulator, and report the results,
	including screenshots (golden files) and logs from `adb logcat`.

	In the following architecture diagram:

	> <a
	> href="https://github.com/flutter/flutter/wiki/The-Engine-architecture"
	> title="The Engine architecture"> <img
	> width="300"
	> alt="Anatomy of a Flutter app"
	> src="https://raw.githubusercontent.com/flutter/engine/main/docs/app_anatomy.svg"
	> /> </a>

	- The Dart app is represented by [`lib/main.dart`](../lib/main.dart).
	- There is no framework code.
	- `dart:ui` and the engine are the same as in a real app.
	- Android-specific application code is in this directory
	([`android/`](./)).
	- The runner is a _custom_ test runner,
	[`run_android_tests.dart`](bin/run_android_tests.dart).

	### Scope of Testing

	The tests in this package are end-to-end tests that _specifically_ exercise the
	engine and Android-specific embedding code. They are not [unit tests][1] for the
	engine or the framework, and are not designed to test the behavior of the
	framework or specific plugins, but rather to simulate the use of a framework
	or plugins downstream of the engine.

	In other words, we test "does the engine work on Android?" without a dependency
	on either the Flutter framework, Flutter tooling, or any specific plugins.

	[1]: ../../../shell/platform/android

	### Golden Comparisons

	Many of the Android-specific interactions with the engine are visual, such as
	[external textures](https://api.flutter.dev/flutter/widgets/Texture-class.html)
	or [platform views](https://docs.flutter.dev/platform-integration/android/platform-views),
	and as such, the tests in this package use golden screenshot file comparisons to
	verify the correctness of the engine's output.

	For example, in [`ExternalTextureTests_testMediaSurface`](https://flutter-engine-gold.skia.org/search?corpus=flutter-engine&include_ignored=false&left_filter=name%3DExternalTextureTests_testMediaSurface&max_rgba=0&min_rgba=0&negative=true&not_at_head=false&positive=true&reference_image_required=false&right_filter=&sort=descending&untriaged=true), a [video](app/src/main/assets/sample.mp4) is converted to an external texture and displayed in a Flutter app. The test takes a screenshot of the app and compares it to a golden file:

	<img
	alt="Two pictures, the top one Flutter and the bottom Android"
	src="https://github.com/flutter/flutter/assets/168174/e2c34b88-d03d-4732-87e4-a86c97d006c5"
	width="300"
	/>

	_The top picture is the Flutter app, and the bottom picture is just Android._

	See also:

	- [`tools/compare_goldens`](../../../tools/compare_goldens).

	## Prerequisites

	If you've never worked in the `flutter/engine` repository before, you will
	need to setup a development environment that is quite different from a typical
	Flutter app or even working on the Flutter framework. It will take roughly
	30 minutes to an hour to setup an environment, depending on your familiarity
	with the tools and the speed of your internet connection.

	See also:

	- [Setting up the Engine](https://github.com/flutter/flutter/wiki/Setting-up-the-Engine-development-environment)
	- [Rebuilding the Tests](#rebuilding-the-tests)

	### Android SDK

	It's highly recommended to use the engine's vendored Android SDK, which once
	you have the engine set up, you can find at
	`$ENGINE/src/flutter/third_party/android_tools/sdk`. Testing or running with
	other versions of the SDK may work, but it's _not guaranteed_, and might have
	different results.

	Consider also placing this directory in the `ANDROID_HOME` environment variable:

	```sh
	export ANDROID_HOME=$ENGINE/src/flutter/third_party/android_tools/sdk
	```

	### Device or Emulator

	The tests in this package require a connected device or emulator to run. The
	device or emulator should be running the same version of Android as the CI
	configuration (there are issues with crashes and other problems on older
	emulators in particular).

	> [!CAUTION]
	>
	> [#144561](https://github.com/flutter/flutter/issues/144561): The emulator
	> vendored in the engine checkout is old and [has a known issue with Vulkan](https://github.com/flutter/flutter/issues/144232).
	>
	> If you're working locally, you can update your copy by running:
	>
	> ```sh
	> $ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager --install emulator
	> ```

	### Additional Dependencies

	While not required, it is strongly recommended to use an IDE such as
	[Android Studio](https://developer.android.com/studio) when contributing to the
	Android side of the test suite, as it will provide a better development
	experience such as code completion, debugging, and more:

	<img
	width="500"
	alt="Screenshot of Android Studio with Autocompletion"
	src="https://github.com/flutter/flutter/assets/168174/79b685f5-8da7-4396-abe6-9baeb29d7ce3"
	/>

	> [!TIP]
	>
	> Android Studio is expected to work _out of the box_ in this directory.
	>
	> If you encounter any issues, please [file a bug](#getting-help).

	## Running the Tests

	The [test runner](../bin/run_android_tests.dart) is a Dart script that installs
	the test app and test suite on a connected device or emulator, runs the tests,
	and reports the results including screenshots (golden files) and logs from
	`adb logcat`.

	From the `$ENGINE/src/flutter` directory, run:

	```sh
	dart ./testing/scenario_app/bin/run_android_tests.dart
	```

	By default when run locally, the test runner:

	- Uses the engine-wide default graphics backend for Android.
	- Uses the last built Android-specific engine artifacts (i.e. `$ENGINE/src/out/android_*/`).
	- Will not diff screenshots, but does save them to a logs directory.

	### Rebuilding the Tests

	If you've made changes to any file in `scenario_app`, incluing the Dart code
	in `lib/` or the Android code in `android/`, you will need to rebuild the
	tests before running them.

	To rebuild the `scenario_app` for the `android_debug_unopt_arm64` variant:

	```sh
	ninja -C out/android_debug_unopt_arm64 scenario_app
	```

	See also:

	- [Compiling the Engine](https://github.com/flutter/flutter/wiki/Compiling-the-engine)

	### Common Options

	A list of options can be found by running:

	```sh
	dart ./testing/scenario_app/bin/run_android_tests.dart --help
	```

	Frequently used options include:

	- `--out-dir`: Which engine artifacts to use (e.g.
	`--out-dir=../out/android_debug_unopt_arm64`).

	- `--logs-dir`: Where to save full `adb logcat` logs and screenshots.

	- `--[no]-enable-impeller`: Enables/disables use of the Impeller graphics
	backend.

	- `--impeller-backend`: Use a specific Impeller backend (e.g.
	`--impeller-backend=opengles`).

	- `--force-surface-producer-surface-texture`: Force the use of `SurfaceTexture`s
	for plugin code that uses `SurfaceProducer`s. This instruments the same code
	path that is used for Android API versions that are <= 28 without requiring
	an older emulator or device.

	- `--smoke-test=<full.class.Name>`: Runs a specific test, instead of all tests.

	### Advanced Options

	When debugging the runner itself, you can use the `--verbose` flag to see more
	detailed output, including additional options such as configuring which binary
	of `adb` to use:

	```sh
	dart ./testing/scenario_app/bin/run_android_tests.dart --help --verbose
	```

	See also:

	- [`bin/utils/options.dart`](../bin/utils/options.dart).

	### CI Configuration

	See [`ci/builders`](../../../ci/builders) and grep for `run_android_tests.dart`:

	```sh
	grep -r run_android_tests.dart ci/builders
	```

	> [!NOTE]
	> The Impeller OpenGLES backend tests are only run on staging (`bringup: true`)
	> and as such are non-blocking. We expect to stabilize and run these tests
	> as part of a wider release of the Impeller OpenGLES backend.

	#### Older Android

	"Older Android" refers to "code paths to support older Android API levels".
	Specifically, these configurations use
	`--force-surface-producer-surface-texture` (see [above](#common-options) for
	details).

	\| Backend \| CI Configuration \| CI History \| Skia Gold \|
	\| ----------------- \| --------------------------- \| ---------------------------------------------- \| ------------------------- \|
	\| Skia \| [`ci/builders`][skia-st-ci] \| [Presubmit][skia-try], [Postsubmit][skia-prod] \| [Skia Gold][skia-st-gold] \|
	\| Impeller OpenGLES \| [`ci/builders`][imp-st-ci] \| [Staging][imp-staging] \| N/A \|

	[skia-try]: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_android_emulator_skia_tests
	[skia-prod]: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20linux_android_emulator_skia_tests
	[imp-staging]: https://ci.chromium.org/ui/p/flutter/builders/staging/Linux%20linux_android_emulator_opengles_tests
	[impeller-try]: https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_android_emulator_tests
	[impeller-prod]: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20linux_android_emulator_tests
	[skia-st-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--no-enable-impeller%22+%22run_android_tests.dart%22+%22--force-surface-producer-surface-texture%22&type=code
	[skia-st-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dtrue%26GraphicsBackend%3Dskia&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia
	[imp-st-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dopengles%22+%22run_android_tests.dart%22+%22--force-surface-producer-surface-texture%22&type=code
	[imp-st-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dtrue%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia

	#### Newer Android

	\| Backend \| CI Configuration \| CI History \| Skia Gold \|
	\| ----------------- \| ---------------------------- \| ------------------------------------------------------ \| ------------------------ \|
	\| Skia \| [`ci/builders`][skia-ci] \| [Presubmit][skia-try], [Postsubmit][skia-prod] \| [Skia Gold][skia-gold] \|
	\| Impeller OpenGLES \| [`ci/builders`][imp-gles-ci] \| [Staging][imp-staging] \| N/A \|
	\| Impeller Vulkan \| [`ci/builders`][imp-vk-ci] \| [Presubmit][impeller-try], [Postsubmit][impeller-prod] \| [Skia Gold][imp-vk-gold] \|

	[skia-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--no-enable-impeller%22+%22run_android_tests.dart%22&type=code
	[skia-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dfalse%26GraphicsBackend%3Dskia&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia
	[imp-gles-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dopengles%22+%22run_android_tests.dart%22&type=code
	[imp-vk-ci]: https://github.com/search?q=repo%3Aflutter%2Fengine+path%3Aci%2Fbuilders+%22--impeller-backend%3Dvulkan%22+%22run_android_tests.dart%22&type=code
	[imp-vk-gold]: https://flutter-engine-gold.skia.org/search?left_filter=ForceSurfaceProducerSurfaceTexture%3Dfalse%26GraphicsBackend%3Dimpeller-vulkan&negative=true&positive=true&right_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia

	## Contributing

	[![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/flutter/flutter/e%3A%20scenario-app)](https://github.com/flutter/flutter/issues?q=is%3Aopen+is%3Aissue+label%3A%22e%3A+scenario-app%22)

	Contributions to this package are welcome, as it is a critical part of the
	engine's test suite.

	### Anatomy of a Test

	A "test" in practice is a combination of roughly 3 components:

	1. An Android _[JUnit test][]_ that configures and launches an Android activity.
	2. An Android _[activity][]_, which simulates the Android side of a plugin or
	platform view.
	3. A Dart _[scenario][]_, which simulates the Flutter side of an application.

	[junit test]: ./app/src/androidTest/java/dev/flutter/scenariosui/DrawSolidBlueScreenTest.java
	[activity]: ./app/src/main/java/dev/flutter/scenarios/ExternalTextureFlutterActivity.java
	[scenario]: ../lib/src/solid_blue.dart

	While every test suite has exactly one JUnit-instrumented class, each test can
	have many activities and scenarios, each with their own configuration, setup,
	and assertions. Not all of this is desirable, but it is the current state of
	the test suite.

	A test might also take a screenshot. See the _Skia Gold_ links in
	[CI Configuration](#ci-configuration) for examples.

	### Project History

	This test suite was [originally written in 2019](https://github.com/flutter/engine/pull/10007)
	with a goal of:

	> \[being\] suitable for embedders to do integration testing with - it has no
	> dependencies on the flutter_tools or framework, and so will not fail/flake
	> based on variances in those downstream.

	Unfortunately, the Android side of the test suite was never fully operational,
	and the tests, even if failing, were accidentally be reported as passing on CI.
	In 2024, as the team got closer to shipping our new graphics backend,
	[Impeller](https://docs.flutter.dev/perf/impeller) on Android, it was clear that
	we needed a reliable test suite for the engine on Android, particularly for
	visual tests around external textures and platform views.

	So, this package was revived and updated to be a (more) reliable test suite for
	the engine on Android. It's by no means complete
	([contributions welcome](#contributing)), but it did successfully catch at least
	one bug that would not have been detected automatically otherwise.

	_Go forth and test the engine on Android!_

	## Troubleshooting

	If you encounter any issues, please [file a bug](#getting-help).

	### My test is failing on CI

	If a test is failing on CI, it's likely that the test is failing locally as
	well. Try the steps in [running the Tests](#running-the-tests) to reproduce the
	failure locally, and then debug the failure as you would any other test. If this
	is your first time working on the engine, you may need to setup a development
	environment first (see [prerequisites](#prerequisites)).

	The test runner makes extensive use of logging and screenshots to help debug
	failures. If you're not sure where to start, try looking at the logs and
	screenshots to see if they provide any clues (you'll need them to file a bug
	anyway).

	`test: Android Scenario App Integration Tests (Impeller/Vulkan)`
	on [LUCI](https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Engine%20Drone/2079486/overview) produces these logs:

	![Screenshot of Logs](https://github.com/flutter/flutter/assets/168174/fed8fae6-bcd2-4e6e-a074-c2c37f34a770)

	The files include a screenshot of each test, and a _full_ logcat log from the
	device (you might find it easier trying the `stdout` of the test first, which
	uses rudimentary log filtering). In the case of multiple runs, the logs are
	prefixed with the test configuration and run attempt.

	### Gradle is failing due to an "not part of the dependency lock state" error

	If you update dependencies in the `app/build.gradle` file, you may encounter an
	error (probably in CI) that looks like:

	```txt
	FAILED: scenario_app/reports/lint-results.xml
	vpython3 ../../flutter/testing/rules/run_gradle.py /b/s/w/ir/cache/builder/src/flutter/testing/scenario_app/android lint --no-daemon -Pflutter_jar=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/flutter.jar -Pout_dir=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app --project-cache-dir=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app/.gradle --gradle-user-home=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app/.gradle

	FAILURE: Build failed with an exception.

	* What went wrong:
	Execution failed for task ':app:checkDebugAarMetadata'.
	> Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
	> Resolved 'org.hamcrest:hamcrest-core:1.3' which is not part of the dependency lock state
	> Resolved 'com.google.code.findbugs:jsr305:2.0.2' which is not part of the dependency lock state
	```

	This is because [`gradle.lockfile`](./app/gradle.lockfile) is out of date. To
	update it, run:

	```sh
	cd testing/scenario_app/android
	$ENGINE_SRC/flutter/third_party/gradle/bin/gradle app:dependencies --write-locks
	```

	## Getting Help

	To suggest changes, or highlight problems, please [file an issue](https://github.com/flutter/flutter/issues/new?labels=e:%20scenario-app,engine,platform-android,fyi-android,team-engine).

	If you're not sure where to start, or need help debugging or contributing, you
	can also reach out to [`hackers-android`](https://discord.com/channels/608014603317936148/846507907876257822) on Discord, or the Flutter engine team internally. There is
	no full-time maintainer of this package, so be prepared to do some of the
	legwork yourself.