Add codesign keychain to search list.

update 04/18:
Another day of research and I finally tracked down EVERY single bit of every root causes of flakes on codesign runs. One of the major problems identified and solved was that, we now add our keychain to the search list.

The latest version has been thoroughly tested on 10+ led runs at different times, (>60 subbuilds in total?) and they all passed. I have 99.9 percent confidence that we will not see a codesign flake anymore. led runs are triggered with `led get-build 8784446248215539137 | led edit-recipe-bundle | led edit-system -p 10 | led launch`. Some example successful runs are:

update 04/17:
Move all keychain setup commands into bash script to run in the same ssh session.
The passes were green in the morning, but started to turn red later in the day. One example failure was

side note, sometimes I see
"WARNING: This build ran on a canary version of LUCI. If you suspect it failed due to infra, retry the build. Next time it may use the non-canary version." but i think this might be unrelated to our use case.

Update 04/16:

I am starting to reproduce the errors in importing certificates. It appears to me that for all the successful runs, the only present certificate should be the flutter certificate. On the other hand, if the device already have two certificates
  1) 5BB6BDEBAA24047669E1A93BCF5CBD2B1E41BB67 "MDM Client Identity Certificate"
  2) CC21BF11AB9925313E54CD4FCCFEB82347E75AFA "Apple Development: Flutter Devicelab (24273VPZX7)"
then the import will be guaranteed to fail. It looks like the success depends on which bot the task is run on.

Update 04/15:
Tuned bash script and test coverage to make them eventually pass. Learnt about set -e with if, local variable, bash expressions etc.

Was trying to test flakiness with 100 led runs and used a script to pipe luci output (not stdout or stderr) of 100 led runs using script command to google sheet. However, the 100 runs were cancelled due to contention for resources. a sample cancelled run looks like:

update 4/14:
wasn't able to reproduce the flake, successful led runs are:

Change-Id: Ic40e62032886ca9cbe77679239fd48ba4744da04
Reviewed-by: Godofredo Contreras <>
Commit-Queue: Xilai Zhang <>
12 files changed
tree: 0ee9f632b9ebf73699fa381655d26dcb325b1b74
  1. infra/
  2. recipe_modules/
  3. recipes/
  4. .gitignore
  5. .style.yapf

Flutter LUCI Recipes

This repository contains Flutter's LUCI recipes. For the LUCI infrastructure config, see flutter/infra. Actual builds can be seen at

Supported repositories roll their .ci.yaml into flutter/infra, which updates what properties builds have. For example, flutter config specifies various dependencies the different tests require, which are then used by the flutter_deps recipe_module No modifications to flutter/infra are required to work on the recipes.


Tricium configurations recipes repo.

Testing locally

Running python test train will execute all of the tests in this repository


Required binary is not found on PATH: cipd when testing locally

You need to ensure depot_tools is installed. See Depot Tools Installation Tutorial