End-to-end tests and test infrastructure for the Flutter engine on Android.
[!IMPORTANT] There are several known issues with this test suite:
Top topics covered in this document include (but are not limited to):
This package simulates a Flutter app that uses the engine (dart:ui) only, in conjunction with Android-specific embedding code that simulates the use of the engine in a real app (such as plugins and platform views).
A custom test runner, run_android_tests.dart, is used to run the tests on a connected device or emulator, and report the results, including screenshots (golden files) and logs from adb logcat.
In the following architecture diagram:
lib/main.dart.dart:ui and the engine are the same as in a real app.android/).run_android_tests.dart.The tests in this package are end-to-end tests that specifically exercise the engine and Android-specific embedding code. They are not unit tests for the engine or the framework, and are not designed to test the behavior of the framework or specific plugins, but rather to simulate the use of a framework or plugins downstream of the engine.
In other words, we test “does the engine work on Android?” without a dependency on either the Flutter framework, Flutter tooling, or any specific plugins.
Many of the Android-specific interactions with the engine are visual, such as external textures or platform views, and as such, the tests in this package use golden screenshot file comparisons to verify the correctness of the engine's output.
For example, in ExternalTextureTests_testMediaSurface, a video is converted to an external texture and displayed in a Flutter app. The test takes a screenshot of the app and compares it to a golden file:
The top picture is the Flutter app, and the bottom picture is just Android.
See also:
If you've never worked in the flutter/engine repository before, you will need to setup a development environment that is quite different from a typical Flutter app or even working on the Flutter framework. It will take roughly 30 minutes to an hour to setup an environment, depending on your familiarity with the tools and the speed of your internet connection.
See also:
It‘s highly recommended to use the engine’s vendored Android SDK, which once you have the engine set up, you can find at $ENGINE/src/flutter/third_party/android_tools/sdk. Testing or running with other versions of the SDK may work, but it's not guaranteed, and might have different results.
Consider also placing this directory in the ANDROID_HOME environment variable:
export ANDROID_HOME=$ENGINE/src/flutter/third_party/android_tools/sdk
The tests in this package require a connected device or emulator to run. The device or emulator should be running the same version of Android as the CI configuration (there are issues with crashes and other problems on older emulators in particular).
[!CAUTION]
#144561: The emulator vendored in the engine checkout is old and has a known issue with Vulkan.
If you're working locally, you can update your copy by running:
$ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager --install emulator
While not required, it is strongly recommended to use an IDE such as Android Studio when contributing to the Android side of the test suite, as it will provide a better development experience such as code completion, debugging, and more:
[!TIP]
Android Studio is expected to work out of the box in this directory.
If you encounter any issues, please file a bug.
The test runner is a Dart script that installs the test app and test suite on a connected device or emulator, runs the tests, and reports the results including screenshots (golden files) and logs from adb logcat.
From the $ENGINE/src/flutter directory, run:
dart ./testing/scenario_app/bin/run_android_tests.dart
By default when run locally, the test runner:
$ENGINE/src/out/android_*/).If you've made changes to any file in scenario_app, incluing the Dart code in lib/ or the Android code in android/, you will need to rebuild the tests before running them.
To rebuild the scenario_app for the android_debug_unopt_arm64 variant:
ninja -C out/android_debug_unopt_arm64 scenario_app
See also:
A list of options can be found by running:
dart ./testing/scenario_app/bin/run_android_tests.dart --help
Frequently used options include:
--out-dir: Which engine artifacts to use (e.g. --out-dir=../out/android_debug_unopt_arm64).
--logs-dir: Where to save full adb logcat logs and screenshots.
--[no]-enable-impeller: Enables/disables use of the Impeller graphics backend.
--impeller-backend: Use a specific Impeller backend (e.g. --impeller-backend=opengles).
--force-surface-producer-surface-texture: Force the use of SurfaceTextures for plugin code that uses SurfaceProducers. This instruments the same code path that is used for Android API versions that are <= 28 without requiring an older emulator or device.
--smoke-test=<full.class.Name>: Runs a specific test, instead of all tests.
When debugging the runner itself, you can use the --verbose flag to see more detailed output, including additional options such as configuring which binary of adb to use:
dart ./testing/scenario_app/bin/run_android_tests.dart --help --verbose
See also:
See ci/builders and grep for run_android_tests.dart:
grep -r run_android_tests.dart ci/builders
[!NOTE] The Impeller OpenGLES backend tests are only run on staging (
bringup: true) and as such are non-blocking. We expect to stabilize and run these tests as part of a wider release of the Impeller OpenGLES backend.
“Older Android” refers to “code paths to support older Android API levels”. Specifically, these configurations use --force-surface-producer-surface-texture (see above for details).
| Backend | CI Configuration | CI History | Skia Gold |
|---|---|---|---|
| Skia | ci/builders | Presubmit, Postsubmit | Skia Gold |
| Impeller OpenGLES | ci/builders | Staging | N/A |
| Backend | CI Configuration | CI History | Skia Gold |
|---|---|---|---|
| Skia | ci/builders | Presubmit, Postsubmit | Skia Gold |
| Impeller OpenGLES | ci/builders | Staging | N/A |
| Impeller Vulkan | ci/builders | Presubmit, Postsubmit | Skia Gold |
Contributions to this package are welcome, as it is a critical part of the engine's test suite.
A “test” in practice is a combination of roughly 3 components:
While every test suite has exactly one JUnit-instrumented class, each test can have many activities and scenarios, each with their own configuration, setup, and assertions. Not all of this is desirable, but it is the current state of the test suite.
A test might also take a screenshot. See the Skia Gold links in CI Configuration for examples.
This test suite was originally written in 2019 with a goal of:
[being] suitable for embedders to do integration testing with - it has no dependencies on the flutter_tools or framework, and so will not fail/flake based on variances in those downstream.
Unfortunately, the Android side of the test suite was never fully operational, and the tests, even if failing, were accidentally be reported as passing on CI. In 2024, as the team got closer to shipping our new graphics backend, Impeller on Android, it was clear that we needed a reliable test suite for the engine on Android, particularly for visual tests around external textures and platform views.
So, this package was revived and updated to be a (more) reliable test suite for the engine on Android. It's by no means complete (contributions welcome), but it did successfully catch at least one bug that would not have been detected automatically otherwise.
Go forth and test the engine on Android!
If you encounter any issues, please file a bug.
If a test is failing on CI, it's likely that the test is failing locally as well. Try the steps in running the Tests to reproduce the failure locally, and then debug the failure as you would any other test. If this is your first time working on the engine, you may need to setup a development environment first (see prerequisites).
The test runner makes extensive use of logging and screenshots to help debug failures. If you‘re not sure where to start, try looking at the logs and screenshots to see if they provide any clues (you’ll need them to file a bug anyway).
test: Android Scenario App Integration Tests (Impeller/Vulkan) on LUCI produces these logs:
The files include a screenshot of each test, and a full logcat log from the device (you might find it easier trying the stdout of the test first, which uses rudimentary log filtering). In the case of multiple runs, the logs are prefixed with the test configuration and run attempt.
If you update dependencies in the app/build.gradle file, you may encounter an error (probably in CI) that looks like:
FAILED: scenario_app/reports/lint-results.xml
vpython3 ../../flutter/testing/rules/run_gradle.py /b/s/w/ir/cache/builder/src/flutter/testing/scenario_app/android lint --no-daemon -Pflutter_jar=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/flutter.jar -Pout_dir=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app --project-cache-dir=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app/.gradle --gradle-user-home=/b/s/w/ir/cache/builder/src/out/android_emulator_skia_debug_x64/scenario_app/.gradle
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:checkDebugAarMetadata'.
> Could not resolve all files for configuration ':app:debugRuntimeClasspath'.
> Resolved 'org.hamcrest:hamcrest-core:1.3' which is not part of the dependency lock state
> Resolved 'com.google.code.findbugs:jsr305:2.0.2' which is not part of the dependency lock state
This is because gradle.lockfile is out of date. To update it, run:
cd testing/scenario_app/android $ENGINE_SRC/flutter/third_party/gradle/bin/gradle app:dependencies --write-locks
To suggest changes, or highlight problems, please file an issue.
If you're not sure where to start, or need help debugging or contributing, you can also reach out to hackers-android on Discord, or the Flutter engine team internally. There is no full-time maintainer of this package, so be prepared to do some of the legwork yourself.