[devicelab] Refresh documentation (#80064)
diff --git a/dev/devicelab/README.md b/dev/devicelab/README.md
index 09adcf5..a101af6 100644
--- a/dev/devicelab/README.md
+++ b/dev/devicelab/README.md
@@ -7,23 +7,27 @@
for testing, this document refers to them as "tests".
Current statuses for the devicelab are available at
-https://flutter-dashboard.appspot.com. See [dashboard user guide](https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md)
+https://flutter-dashboard.appspot.com/#/build. See [dashboard user guide](https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md)
for information on using the dashboards.
-## How the DeviceLab runs tasks
+## Table of Contents
+* [How the DeviceLab runs tests](#how-the-devicelab-runs-tests)
+* [Running tests locally](#running-tests-locally)
+* [Writing tests](#writing-tests)
+* [Adding tests to continuous integration](#adding-tests-to-continuous-integration)
+* [Adding tests to presubmit](#adding-tests-to-presubmit)
-The DeviceLab devices continuously ask Flutter's continuous integration system
-[Cocoon](https://github.com/flutter/cocoon) for tasks to run. When Cocoon has a
-task that is suitable for the device (e.g. Android test), it reserves that
-task for the device. See [manifest.yaml](manifest.yaml) for more information on
-the information used for scheduling tasks.
-1. If the task succeeds, the test runner reports the success to Cocoon. The dashboards
-will show that task in green.
-2. If the task fails, the test runner reports the failure to the server. Cocoon
-increments the run attempt counter and puts the task back in the pool of available
-tasks. If a task does not succeed after a certain number of attempts (as of this writing the limit is 2),
-the task is marked as failed and is displayed using a red color on the dashboard.
+## How the DeviceLab runs tests
+
+DeviceLab tests are run against physical devices in Flutter's lab (the "DeviceLab").
+
+Tasks specify the type of device they are to run on (`linux_android`, `mac_ios`, `mac_android`, `windows_android`, etc).
+When a device in the lab is free, it will pickup tasks that need to be completed.
+
+1. If the task succeeds, the test runner reports the success and uploads its performance metrics to Flutter's infrastructure. Not
+all tasks record performance metrics.
+2. If the task fails, the test runner reports the failure to Flutter's infrastructure and no performance metrics are collected
## Running tests locally
@@ -46,30 +50,18 @@
Notably, it will start and stop Gradle, for instance.
-### Running all tests
-
-To run all tests defined in `manifest.yaml`, use option `-a` (`--all`):
-
-```sh
-../../bin/cache/dart-sdk/bin/dart bin/run.dart -a
-```
-
-This defaults to only running tests supported by your host device's platform
-(`--match-host-platform`) and exiting after the first failure (`--exit`).
-
### Running specific tests
To run a test, use option `-t` (`--task`):
```sh
# from the .../flutter/dev/devicelab directory
-../../bin/cache/dart-sdk/bin/dart bin/run.dart -t {NAME_OR_PATH_OF_TEST}
+../../bin/cache/dart-sdk/bin/dart bin/test_runner.dart test -t {NAME_OR_PATH_OF_TEST}
```
Where `NAME_OR_PATH_OF_TEST` can be either of:
-- the _name_ of a task, which you can find in the `manifest.yaml` file in this
- directory. Example: `complex_layout__start_up`.
+- the _name_ of a task, which is a file's basename in `bin/tasks`. Example: `complex_layout__start_up`.
- the path to a Dart _file_ corresponding to a task, which resides in `bin/tasks`.
Tip: most shells support path auto-completion using the Tab key. Example:
`bin/tasks/complex_layout__start_up.dart`.
@@ -80,14 +72,6 @@
../../bin/cache/dart-sdk/bin/dart bin/run.dart -t test1 -t test2 -t test3
```
-To run tests from a specific stage, use option `-s` (`--stage`).
-Currently, there are only three stages defined, `devicelab`,
-`devicelab_ios` and `devicelab_win`.
-
-```sh
-../../bin/cache/dart-sdk/bin/dart bin/run.dart -s {NAME_OF_STAGE}
-```
-
### Running tests against a local engine build
To run device lab tests against a local engine build, pass the appropriate
@@ -211,31 +195,36 @@
service protocol. This ensures that tasks do not interfere with each other and
lets the CI system time out and clean up tasks that get stuck.
-## Adding tests to the CI environment
+## Adding tests to continuous integration
-The `manifest.yaml` file describes a subset of tests we run in the CI. To add
-your test edit `manifest.yaml` and add the following in the "tasks" dictionary:
+Host only tests should be added to `flutter_tools`.
-```
- {NAME_OF_TEST}:
- description: {DESCRIPTION}
- stage: {STAGE}
- required_agent_capabilities: {CAPABILITIES}
-```
+There are several PRs needed to add a DeviceLab task to CI.
-Where:
-
-- `{NAME_OF_TEST}` is the name of your test that also matches the name of the
+_TASK_- the name of your test that also matches the name of the
file in `bin/tasks` without the `.dart` extension.
-- `{DESCRIPTION}` is the plain English description of your test that helps
- others understand what this test is testing.
-- `{STAGE}` is `devicelab` if you want to run on Android, or `devicelab_ios` if
- you want to run on iOS.
-- `{CAPABILITIES}` is an array that lists the capabilities required of
- the test agent (the computer that runs the test) to run your test. As of writing,
- the available capabilities are: `linux`, `linux/android`, `linux-vm`,
- `mac`, `mac/ios`, `mac/iphonexs`, `mac/ios32`, `mac-catalina/ios`,
- `mac-catalina/android`, `ios/gl-render-image`, `windows`, `windows/android`.
+
+1. Add prod builder to [flutter/infra devicelab_config.star](https://github.com/flutter/infra/blob/master/config/devicelab_config.star)
+ - Example PR: https://github.com/flutter/infra/pull/401/files
+ - This will need to soak for 15 minutes after merged to propagate (should show up in [LUCI console[(https://ci.chromium.org/p/flutter/g/devicelab/console))
+ - There are various lists for the different testbeds a test can run on
+2. Add task to [flutter/flutter prod_builders.json](https://github.com/flutter/flutter/blob/master/dev/prod_builders.json)
+ - Example PR: https://github.com/flutter/flutter/pull/79913/files
+ - Set `flaky: true` for validation before blocking tree
+3. After 10 green runs, update [flutter/flutter prod_builders.json](https://github.com/flutter/flutter/blob/master/dev/prod_builders.json) to `flaky:false`
If your test needs to run on multiple operating systems, create a separate test
for each operating system.
+
+## Adding tests to presubmit
+
+Flutter's DeviceLab does not currently have capacity to run tests against physical devices in presubmit.
+
+Note that DeviceLab tests should generally require a tethered device. If you are adding host-only tests, considering adding your test to `packages/flutter_tools/test/integration.shard`. Example: https://github.com/flutter/flutter/pull/73577/files"
+
+1. Add try builder to [flutter/infra devicelab_config.star](https://github.com/flutter/infra/blob/master/config/devicelab_config.star)
+ - Example PR: https://github.com/flutter/infra/pull/401/files
+ - This will need to soak for 15 minutes after merged to propagate
+ - There are various lists for the different testbeds a test can run on
+2. Add task to [flutter/flutter try_builders.json](https://github.com/flutter/flutter/blob/master/dev/try_builders.json)
+ - Example PR: https://github.com/flutter/flutter/pull/79913/files
