Status: final
Owners: chinmay
Description: Travis reported failures on builds
Component: flutter repository
Date/time: 2016-11-07 21:30
Duration: 16h 45m
User impact: Flutter team members were unable to merge new PRs. Users would have been unable to run flutter tests if they upgraded, though we did not receive complaints during the outage.
A change to package:args is committed (591f9c) that introduces a bug whereby run()
no longer returns the value returned by the command.
15:11 The change to package:args is merged into the args repository.
16:27 Dart package:args tag 0.13.6+1 is cut -- and shortly after is pushed to pub <START OF OUTAGE>
21:36 ianh reports that Travis is upset and all PRs are failing
2016-11-08
07:52 danrubel reports that Travis is still failing
11:03 chinmaygarde reports he’s facing the same breakage in his pending PR
11:07 Issue is reproduced locally. chinmaygarde, jsimmons and danrubel begin looking for the root cause of the breakage.
13:08 Root cause of outage identified as a new version of package:args that Flutter picked up whereby run()
no longer returns the value returned by the command (so we couldn’t get accurate exit codes).
13:19 Flutter PR #6765 sent to pin Flutter to a known good version of package:args
13:42 fb3bf7a identified as root cause of the internal breakage.
14:15 Fix lands. <END OF OUTAGE>
A bug was introduced in package:args that was picked up by Flutter. Flutter was vulnerable to this bug because our external dependencies have open-ended version constraints, so the stability of our codebase is not hermetic. This was an intentional choice; we have experienced this failure mode previously, and have been running on the basis that we are not yet stable enough to deal with the costs of being hermetic.
Action Item | Owner | Tracking bug | Notes |
---|---|---|---|
Pin our external Dart dependencies to specific versions to ensure that our public stability is hermetic. | chinmay | #6767 |
Action Item | Owner | Tracking bug | Notes |
---|---|---|---|
We should have a continuous monitoring bot that tries to run all our tests | ianh | #6777 |
None.
None.
Action Item | Owner | Tracking bug | Notes |
---|---|---|---|
Update our package:args dependency to a known good version | danrubel | PR #6575 | Done |
Deploy a forward-rolling bot that goes red if our dependencies release a breaking change, and otherwise updates us to the latest versions of everything. | ianh | #4696 |