update readme for LUCI (#29023) * update readme for LUCI * Update README.md
diff --git a/dev/bots/README.md b/dev/bots/README.md index 5a4bc87..37599cd 100644 --- a/dev/bots/README.md +++ b/dev/bots/README.md
@@ -5,41 +5,37 @@ The results of such builds are viewable at: * https://cirrus-ci.com/github/flutter/flutter/master - Testing done on PRs and submitted changes on GitHub. -* https://build.chromium.org/p/client.flutter/console +* https://ci.chromium.org/p/flutter/ - Additional testing and processing done after changes are submitted. -The Chromium infra bots do not allow forcing new builds from outside -the Google network. Contact @eseidelGoogle or another Google member of -the Flutter team if you need to do that. +The LUCI infra requires permissions to retrigger or schedule builds. Contact +@kf6gpe or another Google member of the Flutter team if you need to do that. -The [Cirrus](https://cirrus-ci.org)-based bots run the -[`test.dart`](test.dart) script for each PR and submission. This does -testing for the tools, for the framework, and (for submitted changes -only) rebuilds and updates the master branch API docs -[staging site](https://master-docs-flutter-io.firebaseapp.com). -For tagged dev and beta builds, it also builds and deploys the gallery -app to the app stores. It is configured by the -[.cirrus.yml](/.cirrus.yml). +The [Cirrus](https://cirrus-ci.org)-based bots run the [`test.dart`](test.dart) +script for each PR and submission. This does testing for the tools, for the +framework, and (for submitted changes only) rebuilds and updates the master +branch API docs [staging site](https://master-docs.flutter.dev/). +For tagged dev and beta builds, it also builds and deploys the gallery app to +the app stores. It is configured by the [.cirrus.yml](/.cirrus.yml). -We also have post-commit testing with actual devices, in what we call -our [devicelab](../dev/devicelab/README.md). +We also have post-commit testing with actual devices, in what we call our +[devicelab](../dev/devicelab/README.md). -## Chromium infra bots +## LUCI (Layered Universal Continuous Intergration) -This part of our infrastructure is broken into two parts. A buildbot -master specified by our -[builders.pyl](https://chromium.googlesource.com/chromium/tools/build.git/+/master/masters/master.client.flutter/builders.pyl) -file, and a [set of -recipes](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter) -which we run on that master. Both of these technologies are highly -specific to Google's Chromium project. We're just borrowing some of -their infrastructure. +A [set of recipes](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter) +are run on Windows, Linux, and Mac machines. The configuration for how many +machines and what kind are managed internally by Google. Contact @kf6gpe or +another Google member of the Flutter team if you suspect changes are needed +there. Both of these technologies are highly specific to the [LUCI](https://github.com/luci) +project, which is the successor to Chromium's infra. We're just borrowing some +of their infrastructure. ### Prerequisites To work on this infrastructure you will need: -- [depot_tools](http://www.chromium.org/developers/how-tos/install-depot-tools) +- [depot_tools](https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_setting_up) - Python package installer: `sudo apt-get install python-pip` - Python coverage package (only needed for `training_simulation`): `sudo pip install coverage` @@ -58,7 +54,8 @@ ### Getting the code -The following will get way more than just recipe code, but it _will_ get the recipe code: +The following will get way more than just recipe code, but it _will_ get the +recipe code: ```bash mkdir chrome_infra @@ -82,7 +79,7 @@ - build/scripts/slave/recipes/flutter/flutter.py - build/scripts/slave/recipes/flutter/engine.py -Recipes are just Python. They are +Recipes are just Python with some limitations on what can be imported. They are [documented](https://github.com/luci/recipes-py/blob/master/doc/user_guide.md) by the [luci/recipes-py github project](https://github.com/luci/recipes-py). @@ -90,33 +87,30 @@ 1. Make your edits (probably to files in `//chrome_infra/build/scripts/slave/recipes/flutter`). -1. Update the tests. Run `build/scripts/slave/recipes.py test train` to update +2. Update the tests. Run `build/scripts/slave/recipes.py test train` to update existing expected output to match the new output. Verify completely new test - cases by altering the `GenTests` method of the recipe. - The recipe is required to have 100% test coverage. -1. Run `build/scripts/slave/recipes.py run flutter/<repo> slavename=<slavename> - mastername=client.flutter buildername=<buildername> buildnumber=1234` where `<repo>` is one - of `flutter` or `engine`, and `slavename` and `buildername` can be looked up - from the *Build Properties* section of a [recent + cases by altering the `GenTests` method of the recipe. The recipe is required + to have 100% test coverage. +3. Run `build/scripts/slave/recipes.py run flutter/<repo> slavename=<slavename> + mastername=client.flutter buildername=<buildername> + buildnumber=1234` where `<repo>` is one of `flutter` or `engine`, and + `slavename` and `buildername` can be looked up from the *Build Properties* + section of a [recent build](https://build.chromium.org/p/client.flutter/one_line_per_build). -1. To submit a CL, you need a local branch first (`git checkout -b [some branch name]`). -1. Upload the patch (`git commit`, `git cl upload`) and send it to someone in +4. To submit a CL, you need a local branch first (`git checkout -b [some branch name]`). +5. Upload the patch (`git commit`, `git cl upload`) and send it to someone in the `recipes/flutter/OWNERS` file for review. -### Editing the client.flutter buildbot master +### The infra config repository -Flutter uses Chromium's fancy -[builders.pyl](https://chromium.googlesource.com/infra/infra/+/master/doc/users/services/buildbot/builders.pyl.md) -master generation system. Chromium hosts 100s (if not 1000s) of buildbot -masters and thus has lots of infrastructure for turning them up and down. -Eventually all of buildbot is planned to be replaced by other infrastructure, -but for now flutter has its own client.flutter master. +The [flutter/infra](https://github.com/flutter/infra) repository contains +configuration files for the dashboard, builder groups, scheduling, and +individual builders. Edits to this may require changes other internal Google +repositories - e.g., to change the operating system or number of machines. If +you want to do that, reach out to @kf6gpe or another member of the Google team. -You would need to edit client.flutter's master in order to add slaves (talk to -@eseidelGoogle), add builder groups, or to change the html layout of -https://build.chromium.org/p/client.flutter. Carefully follow the [builders.pyl -docs](https://chromium.googlesource.com/infra/infra/+/master/doc/users/services/buildbot/builders.pyl.md) -to do so. +Each configuration file in that repository has a link in the top comments to a +schema that describes available properties. ### Future Directions @@ -130,11 +124,13 @@ ### Android Tools -The Android SDK and NDK used by Flutter's Chrome infra bots are stored in Google Cloud. During the build a bot runs the -`download_android_tools.py` script that downloads the required version of the Android SDK into `dev/bots/android_tools`. +The Android SDK and NDK used by Flutter's Chrome infra bots are stored in Google +Cloud. During the build a bot runs the `download_android_tools.py` script that +downloads the required version of the Android SDK into `dev/bots/android_tools`. -To check which components are currently installed, download the current SDK stored in Google Cloud using the -`download_android_tools.py` script, then `dev/bots/android_tools/sdk/tools/bin/sdkmanager --list`. If you find that some +To check which components are currently installed, download the current SDK +stored in Google Cloud using the `download_android_tools.py` script, then +`dev/bots/android_tools/sdk/tools/bin/sdkmanager --list`. If you find that some components need to be updated or installed, follow the steps below: #### How to update Android SDK on Google Cloud Storage @@ -145,13 +141,17 @@ 2. Use the UI to choose the packages you want to install and/or update. -3. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --update`. On Windows, run `sdkmanager.bat` instead. If the - process fails with an error saying that it is unable to move files (Windows makes files and directories read-only - when another process is holding them open), make a copy of the `dev/bots/android_tools/sdk/tools` directory, run - the `sdkmanager.bat` from the copy, and use the `--sdk_root` option pointing at `dev/bots/android_tools/sdk`. +3. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --update`. On Windows, + run `sdkmanager.bat` instead. If the process fails with an error saying that + it is unable to move files (Windows makes files and directories read-only + when another process is holding them open), make a copy of the + `dev/bots/android_tools/sdk/tools` directory, run the `sdkmanager.bat` from + the copy, and use the `--sdk_root` option pointing at + `dev/bots/android_tools/sdk`. -4. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --licenses` and accept the licenses for the newly installed - components. It also helps to run this command a second time and make sure that it prints "All SDK package licenses +4. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --licenses` and accept + the licenses for the newly installed components. It also helps to run this + command a second time and make sure that it prints "All SDK package licenses accepted". 5. Run upload_android_tools.py -t sdk