Document how to read the dashboard (#14577)
* Document how to read the dashboard
* address comments
diff --git a/dev/devicelab/README.md b/dev/devicelab/README.md
index ed93e80..dd3e1c3 100644
--- a/dev/devicelab/README.md
+++ b/dev/devicelab/README.md
@@ -7,15 +7,91 @@
the tests are referred to as "tasks" in the API, but since we primarily use it
for testing, this document refers to them as "tests".
-If you have access to Google's internal network, you can see the continuous
-build results from the master branch at <http://go/flutter-dashboard/build.html>.
-(There is currently no public view of this data, unfortunately.)
+Build results are available at https://flutter-dashboard.appspot.com.
-# Prerequisites
+# Reading the dashboard
-You must set the `ANDROID_HOME` environment variable to run tests on Android. If
-you have a local build of the Flutter engine, then you have a copy of the
-Android SDK at `.../engine/src/third_party/android_tools/sdk`.
+## The build page
+
+The build page is accessible at https://flutter-dashboard.appspot.com/build.html.
+This page reports the health of build servers, called _agents_, and the statuses
+of build tasks.
+
+### Agents
+
+A green agent is considered healthy and ready to receive new tasks to build. A
+red agent is broken and does not receive new tasks.
+
+In the example below, the dashboard shows that the `linux2` agent is broken and
+requires attention. All other agents are healthy.
+
+
+
+### Tasks
+
+The table below the agent statuses displays the statuses of build tasks. Task
+statuses are color-coded. The following statuses are available:
+
+**New task** (light blue): the task is waiting for an agent to pick it up and
+start the build.
+
+**Task is running** (spinning blue): an agent is currently building the task.
+
+**Task succeeded** (green): an agent reported a successful completion of the
+task.
+
+**Task is flaky** (yellow): the task was attempted multiple time, but only the
+latest attempt succeeded (we currently only try twice).
+
+**Task failed** (red): the task failed all of the attempts.
+
+**Task underperformed** (orange): currently not used.
+
+**Task was skipped** (transparent): the task is not scheduled for a build. This
+usually happens when a task is removed from `manifest.yaml` file.
+
+**Task status unknown** (purple): currently not used.
+
+In addition to color-coding, a task may display a question mark. This means
+that the task was marked as flaky manually. The status of such task is ignored
+when considering whether the build is broken or not. For example, if a flaky
+task fails, GitHub will not prevent PR submissions. However, if the latest
+status of a non-flaky task is red, all pending PRs will contain a warning about
+the broken build and recommend caution when submitting.
+
+Legend:
+
+
+
+The example below shows that commit `e122d5d` caused a wide-spread breakage,
+which was fixed by `bdc6f10`. It also shows that Travis, AppVeyor and Chrome
+Infra (left-most tasks) decided to skip building these commits. Hovering over
+a cell will pop up a tooltip containing the name of the broken task. Clicking
+on the cell will open the log file in a new browser tab (only visible to core
+contributors as of today).
+
+
+
+## Why is a task stuck on "new task" status?
+
+The dashboard aggregates build results from multiple build environments,
+including Travis, AppVeyor, Chrome Infra, and devicelab. While devicelab
+tests every commit that goes into the `master` branch, other environments
+may skip some commits. For example, Travis and AppVeyor will only test the
+_last_ commit of a PR that's merged into the `master` branch. Chrome Infra may
+skip commits when they come in too fast.
+
+## How the devicelab runs the tasks
+
+The devicelab agents have a small script installed on them that continuously
+asks the CI server for tasks to run. When the server finds a suitable task for
+an agent it reserves that task for the agent. If the task succeeds, the agent
+reports the success to the server and the dashboard shows that task in green.
+If the task fails, the agent reports the failure to the server, the server
+increments the counter counting the number of attempts it took to run the task
+and puts the task back in the pool of available tasks. If a task does not
+succeed after a certain number of attempts (as of this writing the limit is 2),
+the task is marked as failed and is displayed using red color on the dashboard.
# Running tests locally
@@ -24,6 +100,12 @@
CI environment runs them. These commands are also useful when you need to
reproduce a CI test failure locally.
+## Prerequisites
+
+You must set the `ANDROID_HOME` environment variable to run tests on Android. If
+you have a local build of the Flutter engine, then you have a copy of the
+Android SDK at `.../engine/src/third_party/android_tools/sdk`.
+
To run a test, use option `-t` (`--task`):
```sh
@@ -61,15 +143,6 @@
# Reproducing broken builds locally
-If a commit caused a test to fail,
-[the dashboard](http://go/flutter-dashboard/build.html) (requires access to the
-Google network, sorry) might look something like this:
-
-
-
-The red circle tells you that a test failed. The number inside tells you how
-many times the devicelab attempted to run the test before giving up on it.
-
To reproduce the breakage locally `git checkout` the corresponding Flutter
revision. Note the name of the test that failed. In the example above the
failing test is `flutter_gallery__transition_perf`. This name can be passed to
diff --git a/dev/devicelab/images/agent-statuses.png b/dev/devicelab/images/agent-statuses.png
new file mode 100644
index 0000000..d0b3dc5
--- /dev/null
+++ b/dev/devicelab/images/agent-statuses.png
Binary files differ
diff --git a/dev/devicelab/images/broken-test.png b/dev/devicelab/images/broken-test.png
index 6450f65..cf6db90 100644
--- a/dev/devicelab/images/broken-test.png
+++ b/dev/devicelab/images/broken-test.png
Binary files differ
diff --git a/dev/devicelab/images/legend.png b/dev/devicelab/images/legend.png
new file mode 100644
index 0000000..c9bd9a0
--- /dev/null
+++ b/dev/devicelab/images/legend.png
Binary files differ
