[osx_sdk] Tolerate duplicate iOS runtime mounts
`install runtimes.verify and mount runtime <ver>` (`xcrun simctl runtime
add <dmg>`) sporadically fails with:
D: <NEW-UDID> iOS (26.2 - 23C54) (Unusable - Other Failure: Error
Domain=SimDiskImageErrorDomain Code=5 "Duplicate of <OLD-UDID>" ...)
Root cause (from the logs of Mac Engine Drone/2490338, Mac Production
Engine Drone/1058394 and /962544 - bb 8673724846373371889,
8673723958432320913, 8683257479423019809): the failure always happens in
the second `install runtimes` pass, right after `Cleaning up runtimes
cache` ran `simctl runtime delete all`. The cleanup considers the
runtimes unmounted as soon as `simctl list runtimes` stops listing them
(~16s), but `simctl runtime delete` is asynchronous: the disk image
stays registered in the `Deleting` state (visible in `simctl runtime
list`) for a while after that. Re-adding the identical dmg (~80s after
the delete in those builds) makes simctl register the new image as an
unusable "Duplicate of" the old one and exit 5. The "Duplicate of" text
only ever shows up in the stdout of `simctl runtime add`; it never
appears in `simctl list runtimes` output. mac_toolchain hit the same
problem and handles it by deleting the duplicate and retrying
(crbug.com/460133386).
Fix:
- Root cause: the post-cleanup wait (`_is_runtimes_unmounted`) now also
polls `xcrun simctl runtime list` ("list runtime disk images") and
keeps waiting while any image is `Deleting`, so the dmg is only
re-added once the old image is really gone.
- Tolerance: `verify and mount runtime` captures stdout and stderr and
no longer fails the build directly. If it fails with a "Duplicate of"
error, a nested "handle duplicate runtime <ver>" step waits for
pending deletions, re-runs `simctl list runtimes` and, if the runtime
build is listed, treats the error as benign. Otherwise it deletes the
disk images named in the error output (`simctl runtime delete
<UDID>`), waits for the deletions and retries the mount once; a second
failure is an INFRA_FAILURE. Any other mount error still fails as
before.
- `_is_runtime_mounted` is split into `_lookup_runtime_build_version`
and `_is_runtime_build_listed`; the latter ignores runtimes listed as
unavailable/unusable so a stale unusable duplicate is not mistaken
for a mounted runtime.
- Timeouts waiting for runtime deletion/unmount are now infra failures
(`InfraFailure` instead of `StepFailure`; `retry.basic_wrap` retries
both), so a bot that never finishes deleting an image is not reported
as a test failure.
Tests (recipe_modules/osx_sdk/examples/full.py):
- mac_13_runtime_duplicate_already_mounted: duplicate error, re-list
shows the runtime -> SUCCESS.
- mac_13_runtime_duplicate_retry_succeeds: duplicate error, image still
`Deleting`, runtime not listed -> delete UDIDs, retry mount ->
SUCCESS.
- mac_13_runtime_duplicate_retry_fails: retry also fails ->
INFRA_FAILURE.
- mac_13_runtime_mount_fails: non-duplicate mount error ->
INFRA_FAILURE.
- mac_13_runtime_listed_unavailable_is_mounted: runtime listed as
unavailable is (re)mounted.
- mac_13_explicit_runtime_version_clean: cleanup waits while a disk
image is `Deleting`.
- mac_13_runtime_duplicate_deletion_never_finishes,
mac_13_cleanup_runtime_deletion_never_finishes: the deletion wait
times out -> INFRA_FAILURE.
Bug: https://github.com/flutter/flutter/issues/180141
Change-Id: I1cfe3dd019ff37353e55271893b4bd86ce9dc6f8
Reviewed-on: https://flutter-review.googlesource.com/c/recipes/+/82420
Reviewed-by: Victoria Ashworth <vashworth@google.com>
Commit-Queue: Elijah Okoroh <okorohelijah@google.com>
This repository contains Flutter's LUCI recipes. For the LUCI infrastructure config, see flutter/infra. Builds can be found in the Flutter Dashboard.
Supported repositories roll their .ci.yaml into flutter/infra, which updates what properties builds have. For example, flutter config specifies various dependencies the different tests require, which are then used by the flutter_deps recipe_module No modifications to flutter/infra are required to work on the recipes.
Please follow instructions on Contributing docs to set up your development environment.