Android Hardware Smoke Test

An integration and compatibility smoke test suite designed to verify visual rendering correctness on Android hardware.

Prerequisites & Initial Setup

Because this integration test project follows a minimal-boilerplate pattern, standard binary Gradle wrappers and properties are not committed to the repository.

Before compiling or running any connected JUnit/instrumented tests locally for the first time, simply run the standard Flutter project regeneration command in this directory to restore the missing wrappers:

flutter create --platform=android --no-overwrite .

This will cleanly restore the missing wrapper scripts (gradlew, gradlew.bat) and wrapper configurations without modifying any of the customized build definitions, Java/Kotlin test harnesses, or package sources.

1. Overview & Purpose

The primary objective of the android_hardware_smoke_test is to provide a fully self-contained Android instrumented test suite.

This allows Android hardware manufacturers (OEMs) to run visual regression, performance, and GPU compatibility tests on a precompiled APK directly on their target devices. The test execution requires no Flutter SDK, Dart CLI commands, or host-side orchestration. Feedback is reported directly in standard native Android JUnit test reports.

2. Structural Differences: android_hardware_smoke_test vs. android_engine_test

While both suites verify rendering correctness, they are architected differently to support different workflows:

3. Dual-Mode Architecture

                        ┌──────────────────────────────────────┐
                        │           Test Initiation            │
                        └──────────────────┬───────────────────┘
                                           │
                  ┌────────────────────────┴────────────────────────┐
                  ▼                                                 ▼
       [ On-Device Instrumented ]                          [ Host-Driven Driver ]
    (OEM / Self-Contained Testing)                       (CI / Host-Driven Testing)
                  │                                                 │
                  ▼                                                 ▼
        native Android JUnit                              flutter drive host script
  (FlutterActivityTest / AndroidJUnit4)             (test_driver/driver_test.dart)
                  │                                                 │
                  ▼                                                 ▼
        Native Java Test sends                            Host driver script connects,
     testName over Message Channel                       and requests testName via
                  │                                      driver.requestData()
                  ▼                                                 │
     App renders target state &                                     ▼
   performs local on-device golden                       App renders target state and
   comparison against bundled assets                     returns image bytes over channel
                  │                                                 │
                  ▼                                                 ▼
     App replies status back to                          Host driver script decodes
      Java; JUnit reports result                         image bytes and asserts match

Instrumented On-Device Mode (OEM / Standalone)

• Orchestration: Runs purely on the device under Android AndroidJUnit4 runner.
• Execution: Java JUnit code (FlutterActivityTest.java) launches the main activity and sends the test payload over a JSON message channel. The app (lib/main.dart) renders the widget, performs a local pixel-by-pixel on-device comparison against baseline images bundled within the APK assets, and replies with the status to the Java runner to pass or fail the JUnit assertion.

Host-Driven Driver Mode (CI / Host-Driven)

• Orchestration: Orchestrated by the host PC using flutter drive.
• Execution: The host script (test_driver/driver_test.dart) commands the target app (integration_test/integration_test_wrapper.dart) to transition states. The app captures the repaint boundary, base64-encodes it, and streams the bytes back to the host over a WebSocket channel. The host driver decodes the bytes and asserts visual matches against local repository baselines on the host filesystem.

4. How to Run the Tests

Unlock your connected Android device or emulator and ensure it is active before executing any of these commands.

A. Running via Host Driver (CI / Host-Driven)

This mode is used to execute visual assertions locally on your PC or in CI pipelines, and to manage the local golden baselines.

• Command to run the driver test suite:

  # Execute from the android_hardware_smoke_test root directory
  flutter drive -v \
    --driver=test_driver/driver_test.dart \
    --target=integration_test/integration_test_wrapper.dart \
    --no-dds
  

• Command to capture/update reference golden baselines: Running with UPDATE_GOLDENS=1 writes or overwrites the local PNG baselines under integration_test/goldens/ on the host.

  Because the statically compiled AndroidManifest.xml is the single source of truth, the app will automatically self-report its active backend variant. To capture or update the baseline for a specific graphics variant, simply edit android/app/src/main/AndroidManifest.xml to set the desired io.flutter.embedding.android.ImpellerBackend value, then execute:

  UPDATE_GOLDENS=1 flutter drive -v \
    --driver=test_driver/driver_test.dart \
    --target=integration_test/integration_test_wrapper.dart \
    --no-dds
  

B. Running Instrumented Tests (OEM / Self-Contained)

[!IMPORTANT] Asset Bundling Precondition: Because instrumented tests run completely standalone on the device, they compare pixels against baseline images bundled as read-only assets inside the APK. You must first generate the local baselines under integration_test/goldens/ using the Host-Driven Driver Mode (with UPDATE_GOLDENS=1) before compiling and building the instrumented APK.

• Command to compile and run the native JUnit suite:

  # Execute from the 'android' subdirectory
  cd android
  ./gradlew :app:connectedDebugAndroidTest \
    -Pandroid.testInstrumentationRunnerArguments.class=com.example.android_hardware_smoke_test.FlutterActivityTest \
    -s
  

[!NOTE] Statically Compiled Single Source of Truth: The app's compiled AndroidManifest.xml <meta-data> tag is the single source of truth for the graphics backend configuration under both Instrumented On-Device Mode (OEM) and Host-Driven Driver Mode (CI).

• Instrumented On-Device Mode (OEM): The native Java JUnit harness (FlutterActivityTest.java) reads this value dynamically using the PackageManager API and routes it to Dart.
• Host-Driven Driver Mode (CI / Host): The Dart app queries the native Android embedder via a custom MethodChannel to self-discover its compiled backend and self-reports it to the host test script inside its JSON reply payload, completely eliminating the need for environment variables on the host PC.

To switch the active graphics backend manually for local runs, open android/app/src/main/AndroidManifest.xml and update the io.flutter.embedding.android.ImpellerBackend value:

<!-- Enable Vulkan: -->
<meta-data android:name="io.flutter.embedding.android.ImpellerBackend" android:value="vulkan" />

<!-- Enable OpenGLES: -->
<meta-data android:name="io.flutter.embedding.android.ImpellerBackend" android:value="opengles" />

[!NOTE] Automated HTML Screenshot Embedding (embedTestResultImages): When running the Gradle command above, a custom Kotlin DSL task named embedTestResultImages executes automatically once the tests finish.

It performs the following actions seamlessly:

1. Prevents Gradle from auto-uninstalling the APKs prematurely (via a gradle.properties injection).
2. Queries the device sandbox cache to discover all rendered .png files dynamically.
3. Streams the raw binary images directly onto the host PC using zero-copy ADB piping.
4. Dynamically parses the generated HTML reports and injects Alternative <img> elements right next to the test outcome table cells.
5. Executes a manual adb uninstall cleanup to leave the target device perfectly clean.

Once finished, open app/build/reports/androidTests/connected/debug/index.html to view the interactive report with all rendering result snapshots embedded natively!

C. Manual Command-Line Debugging (Without Gradle Orchestration)

If you prefer to bypass Gradle entirely for custom debugging, you can manually build, install, and run the instrumentation using raw adb shell calls:

1. Build and Install the packages manually:

   # Run from the 'android' subdirectory
   cd android
   ./gradlew installDebug installDebugAndroidTest
   

2. Manually launch the native Android instrumentation test:

   adb shell am instrument -w \
     -e class com.example.android_hardware_smoke_test.FlutterActivityTest \
     com.example.android_hardware_smoke_test.test/androidx.test.runner.AndroidJUnitRunner
   

3. Manually pull the generated snapshot off the device's sandbox: Since the app remains installed during raw adb runs, you can copy the rendering result files manually:

   adb exec-out "run-as com.example.android_hardware_smoke_test cat cache/results/blueRectangleTest.png" \
     > test_driver/results/blueRectangleTest.png
   

D. Running the CI Shard Locally

Since the test suite is registered inside the central repository test orchestrator (dev/bots/test.dart), you can execute the full CI runner pipeline locally using standard dev-bot scripts:

# Run the Vulkan graphics backend shard locally
SHARD=android_hardware_smoke_vulkan_tests bin/cache/dart-sdk/bin/dart dev/bots/test.dart

# Run the OpenGLES graphics backend shard locally
SHARD=android_hardware_smoke_opengles_tests bin/cache/dart-sdk/bin/dart dev/bots/test.dart