Scenario App: Android Tests and Test Runner
End-to-end tests and test infrastructure for the Flutter engine on Android.
[!IMPORTANT] There are several known issues with this test suite:
Top topics covered in this document include (but are not limited to):
Introduction
This package simulates a Flutter app that uses the engine (dart:ui) only, in conjunction with Android-specific embedding code that simulates the use of the engine in a real app (such as plugins and platform views).
A custom test runner, run_android_tests.dart, is used to run the tests on a connected device or emulator, and report the results, including screenshots (golden files) and logs from adb logcat.
In the following architecture diagram:
- The Dart app is represented by
lib/main.dart. - There is no framework code.
dart:uiand the engine are the same as in a real app.- Android-specific application code is in this directory (
android/). - The runner is a custom test runner,
run_android_tests.dart.
Scope of Testing
The tests in this package are end-to-end tests that specifically exercise the engine and Android-specific embedding code. They are not unit tests for the engine or the framework, and are not designed to test the behavior of the framework or specific plugins, but rather to simulate the use of a framework or plugins downstream of the engine.
In other words, we test “does the engine work on Android?” without a dependency on either the Flutter framework, Flutter tooling, or any specific plugins.
Golden Comparisons
Many of the Android-specific interactions with the engine are visual, such as external textures or platform views, and as such, the tests in this package use golden screenshot file comparisons to verify the correctness of the engine's output.
For example, in ExternalTextureTests_testMediaSurface, a video is converted to an external texture and displayed in a Flutter app. The test takes a screenshot of the app and compares it to a golden file:
The top picture is the Flutter app, and the bottom picture is just Android.
See also:
Prerequisites
If you've never worked in the flutter/engine repository before, you will need to setup a development environment that is quite different from a typical Flutter app or even working on the Flutter framework. It will take roughly 30 minutes to an hour to setup an environment, depending on your familiarity with the tools and the speed of your internet connection.
See also:
Android SDK
It‘s highly recommended to use the engine’s vendored Android SDK, which once you have the engine set up, you can find at $ENGINE/src/third_party/android_tools/sdk. Testing or running with other versions of the SDK may work, but it's not guaranteed, and might have different results.
Consider also placing this directory in the ANDROID_HOME environment variable:
export ANDROID_HOME=$ENGINE/src/third_party/android_tools/sdk
Device or Emulator
The tests in this package require a connected device or emulator to run. The device or emulator should be running the same version of Android as the CI configuration (there are issues with crashes and other problems on older emulators in particular).
[!CAUTION]
#144561: The emulator vendored in the engine checkout is old and has a known issue with Vulkan.
If you're working locally, you can update your copy by running:
$ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager --install emulator
Additional Dependencies
While not required, it is strongly recommended to use an IDE such as Android Studio when contributing to the Android side of the test suite, as it will provide a better development experience such as code completion, debugging, and more:
[!TIP]
Android Studio is expected to work out of the box in this directory.
If you encounter any issues, please file a bug.
Running the Tests
The test runner is a Dart script that installs the test app and test suite on a connected device or emulator, runs the tests, and reports the results including screenshots (golden files) and logs from adb logcat.
From the $ENGINE/src/flutter directory, run:
dart ./testing/scenario_app/bin/run_android_tests.dart
By default when run locally, the test runner:
- Uses the engine-wide default graphics backend for Android.
- Uses the last built Android-specific engine artifacts (i.e.
$ENGINE/src/out/android_*/). - Will not diff screenshots, but does save them to a logs directory.
Rebuilding the Tests
If you've made changes to any file in scenario_app, incluing the Dart code in lib/ or the Android code in android/, you will need to rebuild the tests before running them.
To rebuild the scenario_app for the android_debug_unopt_arm64 variant:
ninja -C out/android_debug_unopt_arm64 scenario_app
See also:
Common Options
A list of options can be found by running:
dart ./testing/scenario_app/bin/run_android_tests.dart --help
Frequently used options include:
--out-dir: Which engine artifacts to use (e.g.--out-dir=../out/android_debug_unopt_arm64).--logs-dir: Where to save fulladb logcatlogs and screenshots.--[no]-enable-impeller: Enables/disables use of the Impeller graphics backend.--impeller-backend: Use a specific Impeller backend (e.g.--impeller-backend=opengles).--force-surface-producer-surface-texture: Force the use ofSurfaceTextures for plugin code that usesSurfaceProducers. This instruments the same code path that is used for Android API versions that are <= 28 without requiring an older emulator or device.--smoke-test=<full.class.Name>: Runs a specific test, instead of all tests.
Advanced Options
When debugging the runner itself, you can use the --verbose flag to see more detailed output, including additional options such as configuring which binary of adb to use:
dart ./testing/scenario_app/bin/run_android_tests.dart --help --verbose
See also:
CI Configuration
See ci/builders and grep for run_android_tests.dart:
grep -r run_android_tests.dart ci/builders
[!NOTE] The Impeller OpenGLES backend tests are only run on staging (
bringup: true) and as such are non-blocking. We expect to stabilize and run these tests as part of a wider release of the Impeller OpenGLES backend.
Older Android
“Older Android” refers to “code paths to support older Android API levels”. Specifically, these configurations use --force-surface-producer-surface-texture (see above for details).
| Backend | CI Configuration | CI History | Skia Gold |
|---|---|---|---|
| Skia | ci/builders | Presubmit, Postsubmit | Skia Gold |
| Impeller OpenGLES | ci/builders | Staging | N/A |
Newer Android
| Backend | CI Configuration | CI History | Skia Gold |
|---|---|---|---|
| Skia | ci/builders | Presubmit, Postsubmit | Skia Gold |
| Impeller OpenGLES | ci/builders | Staging | N/A |
| Impeller Vulkan | ci/builders | Presubmit, Postsubmit | Skia Gold |
Contributing
Contributions to this package are welcome, as it is a critical part of the engine's test suite.
Anatomy of a Test
A “test” in practice is a combination of roughly 3 components:
- An Android JUnit test that configures and launches an Android activity.
- An Android activity, which simulates the Android side of a plugin or platform view.
- A Dart scenario, which simulates the Flutter side of an application.
While every test suite has exactly one JUnit-instrumented class, each test can have many activities and scenarios, each with their own configuration, setup, and assertions. Not all of this is desirable, but it is the current state of the test suite.
A test might also take a screenshot. See the Skia Gold links in CI Configuration for examples.
Project History
This test suite was originally written in 2019 with a goal of:
[being] suitable for embedders to do integration testing with - it has no dependencies on the flutter_tools or framework, and so will not fail/flake based on variances in those downstream.
Unfortunately, the Android side of the test suite was never fully operational, and the tests, even if failing, were accidentally be reported as passing on CI. In 2024, as the team got closer to shipping our new graphics backend, Impeller on Android, it was clear that we needed a reliable test suite for the engine on Android, particularly for visual tests around external textures and platform views.
So, this package was revived and updated to be a (more) reliable test suite for the engine on Android. It's by no means complete (contributions welcome), but it did successfully catch at least one bug that would not have been detected automatically otherwise.
Go forth and test the engine on Android!
Troubleshooting
If you encounter any issues, please file a bug.
My test is failing on CI
If a test is failing on CI, it's likely that the test is failing locally as well. Try the steps in running the Tests to reproduce the failure locally, and then debug the failure as you would any other test. If this is your first time working on the engine, you may need to setup a development environment first (see prerequisites).
The test runner makes extensive use of logging and screenshots to help debug failures. If you‘re not sure where to start, try looking at the logs and screenshots to see if they provide any clues (you’ll need them to file a bug anyway).
test: Android Scenario App Integration Tests (Impeller/Vulkan) on LUCI produces these logs:
The files include a screenshot of each test, and a full logcat log from the device (you might find it easier trying the stdout of the test first, which uses rudimentary log filtering). In the case of multiple runs, the logs are prefixed with the test configuration and run attempt.
Gradle is failing due to an “not part of the dependency lock state” error
If you update dependencies in the app/build.gradle file, you may encounter an error (probably in CI) that looks like:
FAILED: scenario_app/reports/lint-results.xml
vpython3 ../../flutter/testing/rules/run_gradle.py /b/s/w/ir/cache/builder/src/flutter/testing/scenario_app/android lint --no-daemon -Pflutter_jar=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/flutter.jar -Pout_dir=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app --project-cache-dir=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app/.gradle --gradle-user-home=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app/.gradle
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:checkDebugAarMetadata'.
> Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
> Resolved 'org.hamcrest:hamcrest-core:1.3' which is not part of the dependency lock state
> Resolved 'com.google.code.findbugs:jsr305:2.0.2' which is not part of the dependency lock state
This is because gradle.lockfile is out of date. To update it, run:
cd testing/scenario_app/android $ENGINE_SRC/third_party/gradle/bin/gradle app:dependencies --write-locks
Getting Help
To suggest changes, or highlight problems, please file an issue.
If you're not sure where to start, or need help debugging or contributing, you can also reach out to hackers-android on Discord, or the Flutter engine team internally. There is no full-time maintainer of this package, so be prepared to do some of the legwork yourself.