docs/infra/Ci-Best-Practices.md - mirrors/flutter - Git at Google

 # Flutter's CI Best Practices

 [flutter.dev/to/ci-yaml](https://flutter.dev/to/ci-yaml)

 This is a supplemental resource to provide best practices for using Flutter's
 internal CI/CD system. It is not a comprehensive guide, but rather a collection of tips and tricks to be as efficient as possible.

 ---

 Table of Contents

 - [Flutter's CI Best Practices](#flutters-ci-best-practices)
   - ["This branch is out-of-date" is not (always) a failure](#this-branch-is-out-of-date-is-not-always-a-failure)
   - [Prefer the `auto-submit` label over pressing "merge"](#prefer-the-auto-submit-label-over-pressing-merge)
   - [Prefer the `revert` label over manual reverts of recent commits](#prefer-the-revert-label-over-manual-reverts-of-recent-commits)
   - [Prefer re-running test suites over re-triggering all tests](#prefer-re-running-test-suites-over-re-triggering-all-tests)
   - [Cost of Adding New Test Targets](#cost-of-adding-new-test-targets)
   - [Cost of Renaming/Resharding Tests](#cost-of-renamingresharding-tests)

 ## "This branch is out-of-date" is not (always) a failure

 <img
   alt="This branch is out-of-date"
   width="600"
   src="https://github.com/user-attachments/assets/0de2ad69-fca0-4c63-81ab-79d1b53162d1" />

 This message is automatically created by GitHub when at least one commit has
 merged into the ancestor branch of the PR. On Flutter's owned repositories, this
 is just a warning, and does not prevent merging.

 There are two edge cases where merging/rebasing is required or recommended:

 1. **When the commit your PR was branched from is causing a test to fail**. If
    a forward fix or revert has since been merged into the ancestor branch, it is
    recommended to rebase your PR to pick up the fix.

 2. **When the commit your PR was branched from is more than a few days old**. If
    the commit is more than a few days old, it is recommended to rebase your PR
    in order to make sure that the tests are run against the latest code.

 ## Prefer the `auto-submit` label over pressing "merge"

 <img
   width="400"
   alt="Adding the auto-submit label"
   src="https://github.com/user-attachments/assets/2b9a3d5a-afb0-4c24-8959-5262ce993beb" />

 The `auto-submit` label is a special label that can be added to a PR to be
 automatically merged (or queued for merging, in `flutter/flutter`'s case) when
 the tests are passing. This is the preferred way to merge PRs.

 If one or more tests fail, and you're confident that the failure is not related
 to your PR (i.e. is a flake), see [prefer re-running test suites over re-triggering all tests](#prefer-re-running-test-suites-over-re-triggering-all-tests).

 ## Prefer the `revert` label over manual reverts of recent commits

 <img
   width="394"
   alt="Adding the revert label"
   src="https://github.com/user-attachments/assets/30fb773e-8b14-4580-94f0-6a650c475469" />

 The `revert` label is a special label that can be added to a PR to revert the
 commit(s) that are causing a CI failure, and is the preferred way to revert
 recently merged commits as it bypasses most testing to get the revert landed as
 fast as possible.

 Before using this label, You'll need to add a comment on the PR:

 ```md
 reason for revert: This breaks the build, see XYZ link.
 ```

 Please coordinate with the Google team that administrates the repository before
 using the `revert` label, as it may not be appropriate to use in all cases, and
 could cause confusion if used incorrectly.

 ## Prefer re-running test suites over re-triggering all tests

 <img
   width="341"
   alt="Failing tests with a re-run button"
   src="https://github.com/user-attachments/assets/0076dd4f-3bc3-4ade-8d71-24b1a5e28272" />

 Each full test run in `flutter/flutter` uses between 100 and 200 virtual
 machines, and takes between 30 and 60 minutes to complete. This is a lot of
 resources, and should be avoided if possible.

 Pushing a new commit, or using the built-in merge button in GitHub, will
 re-trigger all tests in the PR. While sometimes this is necessary, it is a lot
 more expensive than re-running a single test suite or even a few test suites.

 ## Cost of Adding New Test Targets

 The Flutter team has a finite amount of resources to run tests, and adding
 new test targets, while incrementally useful, can add up to a lot of waiting
 time for the team.

 As of 2025/05/23, the presubmit pool for `flutter/flutter` has:

 - [301 Linux VMs](https://chromium-swarm.appspot.com/botlist?c=id&c=task&c=os&c=status&d=asc&f=os%3AUbuntu&f=pool%3Aluci.flutter.try)
 - [131 ARM64 Mac VMs](https://chromium-swarm.appspot.com/botlist?f=cpu%3Aarm64&f=os%3AMac-14&f=pool%3Aluci.flutter.try)
 - [134 Windows VMs](https://chromium-swarm.appspot.com/botlist?f=os%3AWindows-10&f=pool%3Aluci.flutter.try)

 Resources for on-device testing (device lab) are even more limited.

 When you add a new test target, it will consume an entire VM for the time it
 takes to clone the repository, setup the test infrastructure, and run the test
 suite (often between 15 and 30 minutes, but longer for some integration tests).

 While it's important to have a good test suite, consider:

 - Does my test target have to run on multiple platforms?
 - Is there an existing test target that I can add my test to that is not close
   to the timeout limit?

 Consider consulting with the infrastruture team (`team-infra`) before adding a
 large number of new test targets, or to get advice on how to best create new
 test targets.

 ## Cost of Renaming/Resharding Tests

 _This advice only applies to the `flutter/flutter` repository._

 Due to how the test infrastructure works, renaming or resharding tests can be
 very time-consuming, and should be avoided if possible, or carefully coordinated
 when required:

 - New shards must be initially added as `bringup: true`, which means that they
   will not run in presubmit, and do not trigger tree closures in postsubmit.
 - The shards will have to be moved to `bringup: false` once they are
   successfully running, which will require a second PR.
 - Deleting (or renaming) a shard at tip-of-tree (i.e. `master`) will cause the
   test to be silently skipped in all release candidate branches, and will
   require the `.ci.yaml` file to be cherry-picked into each release candidate
   branch to restore the test coverage.

 For example, imagine the following existing test configuration (`.ci.yaml`):

 ```yaml
 targets:
   - name: Linux web_tests_1_2
     shard: web_tests
     subshard: "0"
   - name: Linux web_tests_2_2
     shard: web_tests
     subshard: "1"
 ```

 Behind the scenes, this creates two LUCI builders, one for each test target.

 Now imagine you want to add a third shard (effectively, renaming the targets):

 ```yaml
 targets:
   - name: Linux web_tests_1_3
     shard: web_tests
     subshard: "0"
   - name: Linux web_tests_2_3
     shard: web_tests
     subshard: "1"
   - name: Linux web_tests_3_3
     shard: web_tests
     subshard: "2"
 ```

 This will require the following steps:

 1. Add 3 new shards to the `.ci.yaml` file, and set `bringup: true` for each of them.
 2. Wait for the new shards to be successfully running.
 3. Remove the old shards from the `.ci.yaml` file, and set `bringup: false` for each of them.
 4. Cherry-pick the `.ci.yaml` file into each release candidate branch to restore the test coverage.

 This could require up to 4 PRs, and a lot of coordination with the release team.
	# Flutter's CI Best Practices

	[flutter.dev/to/ci-yaml](https://flutter.dev/to/ci-yaml)

	This is a supplemental resource to provide best practices for using Flutter's
	internal CI/CD system. It is not a comprehensive guide, but rather a collection of tips and tricks to be as efficient as possible.

	---

	Table of Contents

	- [Flutter's CI Best Practices](#flutters-ci-best-practices)
	- ["This branch is out-of-date" is not (always) a failure](#this-branch-is-out-of-date-is-not-always-a-failure)
	- [Prefer the `auto-submit` label over pressing "merge"](#prefer-the-auto-submit-label-over-pressing-merge)
	- [Prefer the `revert` label over manual reverts of recent commits](#prefer-the-revert-label-over-manual-reverts-of-recent-commits)
	- [Prefer re-running test suites over re-triggering all tests](#prefer-re-running-test-suites-over-re-triggering-all-tests)
	- [Cost of Adding New Test Targets](#cost-of-adding-new-test-targets)
	- [Cost of Renaming/Resharding Tests](#cost-of-renamingresharding-tests)

	## "This branch is out-of-date" is not (always) a failure

	<img
	alt="This branch is out-of-date"
	width="600"
	src="https://github.com/user-attachments/assets/0de2ad69-fca0-4c63-81ab-79d1b53162d1" />

	This message is automatically created by GitHub when at least one commit has
	merged into the ancestor branch of the PR. On Flutter's owned repositories, this
	is just a warning, and does not prevent merging.

	There are two edge cases where merging/rebasing is required or recommended:

	1. When the commit your PR was branched from is causing a test to fail. If
	a forward fix or revert has since been merged into the ancestor branch, it is
	recommended to rebase your PR to pick up the fix.

	2. When the commit your PR was branched from is more than a few days old. If
	the commit is more than a few days old, it is recommended to rebase your PR
	in order to make sure that the tests are run against the latest code.

	## Prefer the `auto-submit` label over pressing "merge"

	<img
	width="400"
	alt="Adding the auto-submit label"
	src="https://github.com/user-attachments/assets/2b9a3d5a-afb0-4c24-8959-5262ce993beb" />

	The `auto-submit` label is a special label that can be added to a PR to be
	automatically merged (or queued for merging, in `flutter/flutter`'s case) when
	the tests are passing. This is the preferred way to merge PRs.

	If one or more tests fail, and you're confident that the failure is not related
	to your PR (i.e. is a flake), see [prefer re-running test suites over re-triggering all tests](#prefer-re-running-test-suites-over-re-triggering-all-tests).

	## Prefer the `revert` label over manual reverts of recent commits

	<img
	width="394"
	alt="Adding the revert label"
	src="https://github.com/user-attachments/assets/30fb773e-8b14-4580-94f0-6a650c475469" />

	The `revert` label is a special label that can be added to a PR to revert the
	commit(s) that are causing a CI failure, and is the preferred way to revert
	recently merged commits as it bypasses most testing to get the revert landed as
	fast as possible.

	Before using this label, You'll need to add a comment on the PR:

	```md
	reason for revert: This breaks the build, see XYZ link.
	```

	Please coordinate with the Google team that administrates the repository before
	using the `revert` label, as it may not be appropriate to use in all cases, and
	could cause confusion if used incorrectly.

	## Prefer re-running test suites over re-triggering all tests

	<img
	width="341"
	alt="Failing tests with a re-run button"
	src="https://github.com/user-attachments/assets/0076dd4f-3bc3-4ade-8d71-24b1a5e28272" />

	Each full test run in `flutter/flutter` uses between 100 and 200 virtual
	machines, and takes between 30 and 60 minutes to complete. This is a lot of
	resources, and should be avoided if possible.

	Pushing a new commit, or using the built-in merge button in GitHub, will
	re-trigger all tests in the PR. While sometimes this is necessary, it is a lot
	more expensive than re-running a single test suite or even a few test suites.

	## Cost of Adding New Test Targets

	The Flutter team has a finite amount of resources to run tests, and adding
	new test targets, while incrementally useful, can add up to a lot of waiting
	time for the team.

	As of 2025/05/23, the presubmit pool for `flutter/flutter` has:

	- [301 Linux VMs](https://chromium-swarm.appspot.com/botlist?c=id&c=task&c=os&c=status&d=asc&f=os%3AUbuntu&f=pool%3Aluci.flutter.try)
	- [131 ARM64 Mac VMs](https://chromium-swarm.appspot.com/botlist?f=cpu%3Aarm64&f=os%3AMac-14&f=pool%3Aluci.flutter.try)
	- [134 Windows VMs](https://chromium-swarm.appspot.com/botlist?f=os%3AWindows-10&f=pool%3Aluci.flutter.try)

	Resources for on-device testing (device lab) are even more limited.

	When you add a new test target, it will consume an entire VM for the time it
	takes to clone the repository, setup the test infrastructure, and run the test
	suite (often between 15 and 30 minutes, but longer for some integration tests).

	While it's important to have a good test suite, consider:

	- Does my test target have to run on multiple platforms?
	- Is there an existing test target that I can add my test to that is not close
	to the timeout limit?

	Consider consulting with the infrastruture team (`team-infra`) before adding a
	large number of new test targets, or to get advice on how to best create new
	test targets.

	## Cost of Renaming/Resharding Tests

	_This advice only applies to the `flutter/flutter` repository._

	Due to how the test infrastructure works, renaming or resharding tests can be
	very time-consuming, and should be avoided if possible, or carefully coordinated
	when required:

	- New shards must be initially added as `bringup: true`, which means that they
	will not run in presubmit, and do not trigger tree closures in postsubmit.
	- The shards will have to be moved to `bringup: false` once they are
	successfully running, which will require a second PR.
	- Deleting (or renaming) a shard at tip-of-tree (i.e. `master`) will cause the
	test to be silently skipped in all release candidate branches, and will
	require the `.ci.yaml` file to be cherry-picked into each release candidate
	branch to restore the test coverage.

	For example, imagine the following existing test configuration (`.ci.yaml`):

	```yaml
	targets:
	- name: Linux web_tests_1_2
	shard: web_tests
	subshard: "0"
	- name: Linux web_tests_2_2
	shard: web_tests
	subshard: "1"
	```

	Behind the scenes, this creates two LUCI builders, one for each test target.

	Now imagine you want to add a third shard (effectively, renaming the targets):

	```yaml
	targets:
	- name: Linux web_tests_1_3
	shard: web_tests
	subshard: "0"
	- name: Linux web_tests_2_3
	shard: web_tests
	subshard: "1"
	- name: Linux web_tests_3_3
	shard: web_tests
	subshard: "2"
	```

	This will require the following steps:

	1. Add 3 new shards to the `.ci.yaml` file, and set `bringup: true` for each of them.
	2. Wait for the new shards to be successfully running.
	3. Remove the old shards from the `.ci.yaml` file, and set `bringup: false` for each of them.
	4. Cherry-pick the `.ci.yaml` file into each release candidate branch to restore the test coverage.

	This could require up to 4 PRs, and a lot of coordination with the release team.