tree 0ee9f632b9ebf73699fa381655d26dcb325b1b74
parent e341b0a4e03e1bbd4b8bb412b7c669e546fbb066
author Xilai Zhang <xilaizhang@google.com> 1681876748 +0000
committer CQ Bot Account <flutter-scoped@luci-project-accounts.iam.gserviceaccount.com> 1681876748 +0000

Add codesign keychain to search list.

update 04/18:
Another day of research and I finally tracked down EVERY single bit of every root causes of flakes on codesign runs. One of the major problems identified and solved was that, we now add our keychain to the search list.

The latest version has been thoroughly tested on 10+ led runs at different times, (>60 subbuilds in total?) and they all passed. I have 99.9 percent confidence that we will not see a codesign flake anymore. led runs are triggered with `led get-build 8784446248215539137 | led edit-recipe-bundle | led edit-system -p 10 | led launch`. Some example successful runs are:

https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/xilaizhang_google.com/ba79a610f55ba0d9afd53534d72bc02cd90bd8416a7f7ed7589242039a6361f2/+/build.proto?server=chromium-swarm.appspot.com

https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/xilaizhang_google.com/771e1a09798e2abfa9b548a9b1ee899573129ea67f7cbc6b0e379f2e31ea578a/+/build.proto?server=chromium-swarm.appspot.com

update 04/17:
Move all keychain setup commands into bash script to run in the same ssh session.
The passes were green in the morning, but started to turn red later in the day. One example failure was
https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/xilaizhang_google.com/b789c3878bda578f21b55983d5ec9dd6d7029e0182e60aafdfd83a385535a9a6/+/build.proto?server=chromium-swarm.appspot.com

side note, sometimes I see
"WARNING: This build ran on a canary version of LUCI. If you suspect it failed due to infra, retry the build. Next time it may use the non-canary version." but i think this might be unrelated to our use case.

Update 04/16:

I am starting to reproduce the errors in importing certificates. It appears to me that for all the successful runs, the only present certificate should be the flutter certificate. On the other hand, if the device already have two certificates
  1) 5BB6BDEBAA24047669E1A93BCF5CBD2B1E41BB67 "MDM Client Identity Certificate"
  2) CC21BF11AB9925313E54CD4FCCFEB82347E75AFA "Apple Development: Flutter Devicelab (24273VPZX7)"
then the import will be guaranteed to fail. It looks like the success depends on which bot the task is run on.


https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/xilaizhang_google.com/f8de2c9b8bc1ba038fb34202c7d4557ac5f41ceb1d1075b139f5c50cf14c0e5a/+/build.proto?server=chromium-swarm.appspot.com

https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/xilaizhang_google.com/fe251512a89251551e70c3e54ba2993e9e7eb27a874c0b806b6ebc9315fd6f6d/+/build.proto?server=chromium-swarm.appspot.com

Update 04/15:
Tuned bash script and test coverage to make them eventually pass. Learnt about set -e with if, local variable, bash expressions etc.

Was trying to test flakiness with 100 led runs and used a script to pipe luci output (not stdout or stderr) of 100 led runs using script command to google sheet. However, the 100 runs were cancelled due to contention for resources. a sample cancelled run looks like:

https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/xilaizhang_google.com/59b48935638603ac39c73aa8e8ccb69943f10423f9799026e47daae83d98736c/+/build.proto?server=chromium-swarm.appspot.com

update 4/14:
wasn't able to reproduce the flake, successful led runs are:
https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/xilaizhang_google.com/d57db265bec15c048ed82199f0d8f77a0b9e67ba109577c02d57dcd747401227/+/build.proto?server=chromium-swarm.appspot.com

https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/xilaizhang_google.com/20efcb900545754ea92434f7ba64339f3571104089dadb51e564a0a7721d7768/+/build.proto?server=chromium-swarm.appspot.com

https://ci.chromium.org/raw/build/logs.chromium.org/flutter/led/xilaizhang_google.com/4e8f7c2696f7a39ac4721198d83896f00dea821814cd3cd3918ca7895a39817b/+/build.proto?server=chromium-swarm.appspot.com

Change-Id: Ic40e62032886ca9cbe77679239fd48ba4744da04
Reviewed-on: https://flutter-review.googlesource.com/c/recipes/+/41900
Reviewed-by: Godofredo Contreras <godofredoc@google.com>
Commit-Queue: Xilai Zhang <xilaizhang@google.com>