Flutter's CI Best Practices

flutter.dev/to/ci-yaml

This is a supplemental resource to provide best practices for using Flutter's internal CI/CD system. It is not a comprehensive guide, but rather a collection of tips and tricks to be as efficient as possible.

---

Table of Contents

• Flutter's CI Best Practices
  • “This branch is out-of-date” is not (always) a failure
  • Prefer the auto-submit label over pressing “merge”
  • Prefer the revert label over manual reverts of recent commits
  • Prefer re-running test suites over re-triggering all tests
  • Cost of Adding New Test Targets
  • Cost of Renaming/Resharding Tests

“This branch is out-of-date” is not (always) a failure

This message is automatically created by GitHub when at least one commit has merged into the ancestor branch of the PR. On Flutter's owned repositories, this is just a warning, and does not prevent merging.

There are two edge cases where merging/rebasing is required or recommended:

1. When the commit your PR was branched from is causing a test to fail. If a forward fix or revert has since been merged into the ancestor branch, it is recommended to rebase your PR to pick up the fix.

2. When the commit your PR was branched from is more than a few days old. If the commit is more than a few days old, it is recommended to rebase your PR in order to make sure that the tests are run against the latest code.

Prefer the auto-submit label over pressing “merge”

The auto-submit label is a special label that can be added to a PR to be automatically merged (or queued for merging, in flutter/flutter's case) when the tests are passing. This is the preferred way to merge PRs.

If one or more tests fail, and you're confident that the failure is not related to your PR (i.e. is a flake), see prefer re-running test suites over re-triggering all tests.

Prefer the revert label over manual reverts of recent commits

The revert label is a special label that can be added to a PR to revert the commit(s) that are causing a CI failure, and is the preferred way to revert recently merged commits as it bypasses most testing to get the revert landed as fast as possible.

Before using this label, You'll need to add a comment on the PR:

reason for revert: This breaks the build, see XYZ link.

Please coordinate with the Google team that administrates the repository before using the revert label, as it may not be appropriate to use in all cases, and could cause confusion if used incorrectly.

Prefer re-running test suites over re-triggering all tests

Each full test run in flutter/flutter uses between 100 and 200 virtual machines, and takes between 30 and 60 minutes to complete. This is a lot of resources, and should be avoided if possible.

Pushing a new commit, or using the built-in merge button in GitHub, will re-trigger all tests in the PR. While sometimes this is necessary, it is a lot more expensive than re-running a single test suite or even a few test suites.

Cost of Adding New Test Targets

The Flutter team has a finite amount of resources to run tests, and adding new test targets, while incrementally useful, can add up to a lot of waiting time for the team.

As of 2025/05/23, the presubmit pool for flutter/flutter has:

• 301 Linux VMs
• 131 ARM64 Mac VMs
• 134 Windows VMs

Resources for on-device testing (device lab) are even more limited.

When you add a new test target, it will consume an entire VM for the time it takes to clone the repository, setup the test infrastructure, and run the test suite (often between 15 and 30 minutes, but longer for some integration tests).

While it's important to have a good test suite, consider:

• Does my test target have to run on multiple platforms?
• Is there an existing test target that I can add my test to that is not close to the timeout limit?

Consider consulting with the infrastruture team (team-infra) before adding a large number of new test targets, or to get advice on how to best create new test targets.

Cost of Renaming/Resharding Tests

This advice only applies to the flutter/flutter repository.

Due to how the test infrastructure works, renaming or resharding tests can be very time-consuming, and should be avoided if possible, or carefully coordinated when required:

• New shards must be initially added as bringup: true, which means that they will not run in presubmit, and do not trigger tree closures in postsubmit.
• The shards will have to be moved to bringup: false once they are successfully running, which will require a second PR.
• Deleting (or renaming) a shard at tip-of-tree (i.e. master) will cause the test to be silently skipped in all release candidate branches, and will require the .ci.yaml file to be cherry-picked into each release candidate branch to restore the test coverage.

For example, imagine the following existing test configuration (.ci.yaml):

targets:
  - name: Linux web_tests_1_2
    shard: web_tests
    subshard: "0"
  - name: Linux web_tests_2_2
    shard: web_tests
    subshard: "1"

Behind the scenes, this creates two LUCI builders, one for each test target.

Now imagine you want to add a third shard (effectively, renaming the targets):

targets:
  - name: Linux web_tests_1_3
    shard: web_tests
    subshard: "0"
  - name: Linux web_tests_2_3
    shard: web_tests
    subshard: "1"
  - name: Linux web_tests_3_3
    shard: web_tests
    subshard: "2"

This will require the following steps:

1. Add 3 new shards to the .ci.yaml file, and set bringup: true for each of them.
2. Wait for the new shards to be successfully running.
3. Remove the old shards from the .ci.yaml file, and set bringup: false for each of them.
4. Cherry-pick the .ci.yaml file into each release candidate branch to restore the test coverage.

This could require up to 4 PRs, and a lot of coordination with the release team.