The Flutter Engine Build Definition Language describes a build on CI by defining a combination of sub-builds, archives, generators and dependencies. It makes it simple to shard sub-builds by mapping build inputs to workflows, and listing the sub-build-generated artifacts explicitly. The Build Definition Language, Engine Recipes V2 and the generation of artifacts using GN+Ninja set the groundwork for efficient builds with dependency reusability.
Author: Godofredo Contreras (godofredoc)
Go Link: flutter.dev/go/engine-build-definition-language
Created: 01/2023 / Last updated: 04/2023
Engine build definition files using the Build Definition Language can be found in the flutter/engine/ci/builders directory.
The engine orchestrator recipe reads each file in that directory, shards their builds, collects artifacts and uploads them to the Google Cloud Storage bucket.
The .ci.yaml file at the root of the flutter/engine repository puts all the components together. Builds are specified in that file using a property pointing to the build definition file to be used by engine_v2 recipes. Full documentation of the .ci.yaml file format can be found in the Cocoon repository here.
The following is a sample build configuration referencing android_aot_engine.json in the config_name under properties:
- name: Mac mac_android_aot_engine recipe: engine_v2/engine_v2 timeout: 60 properties: config_name: mac_android_aot_engine $flutter/osx_sdk : >- { "sdk_version": "14e222b" }
To keep the build definition language simple the following assumptions were made during its design:
gn configuration step, a ninja build step, followed by self-contained test scripts, and self-contained generator scripts. All the elements are optional allowing to use gn+ninja without generators or generators without gn+ninja.The build configuration is a json file containing a list of builds, tests, generators and archives. The following is an example of an empty configuration file:
{ "builds": [], "tests": [], "generators": { "tasks": [] }, "archives": [ ] }
Note: tests, generators and archives can be omited if empty.
Build configuration files have to be checked into the engine_checkout/ci/builder directory where engine v2 recipes will be reading them from.
Configurations with a single build are supported. Single build configurations are located have to be checked into the engine_checkout/ci/builder/standalone
A configuration file defines a top-level builder that will show up as a column in the Flutter Dashboard.
A build is a dictionary with a gn command, a ninja command, zero or more generator commands, zero or more local tests, zero or more local generators and zero or more output artifacts.
The following is the high level structure of the build component:
{ "archives": [], "drone_dimensions": [], "gclient_variables": {}, "gn": [], "name": "host_debug", "generators": [], "ninja": {}, "tests": [] }
Each build element will be translated to an independent sub-build and its entire out directory will be uploaded to CAS.
gn, ninja, generators and tests properties are optional. Gn and ninja properties can be used without generators or tests. Generators with no gn and ninja properties is also supported.
An archive component is used to tell the recipes which artifacts are generated by the build and where to upload them.
By default the build output is archived to CAS in order to be used as a dependency for global tests. If no CAS archive is required cas_archive": false, needs to be added to the configuration.
{ "name": "host_debug", "base_path": "out/host_debug/zip_archives/", "type": "gcs", "include_paths": [ "out/host_debug/zip_archives/linux-x64/artifacts.zip" ] }
Description of the fields:
name is used to associate the CAS hash to a value that can be referenced later as a dependency of global tests. Name is also used to select the folder from within src/out to upload to CAS. e.g if the build generates src/out/host_debug name must be host_debug.A list of strings with key value pairs separated by an equal sign. These dimensions are used to select the bot where the sub-build will be running.
To find the list of valid keys and values you need to select a bot from the swarming UI. On the dimensions section the left column contains the keys and the right column contains the allowed values. If multiple values are allowed for a key they are separated using | (pipe symbol).
"drone_dimensions": [ "device_type=none", "os=Linux" ]
In the previous example, the build containing this drone_dimensions component will run on a bot with a Linux OS that does not have any devices attached to it.
Drone dimensions accept values separates by | to specify more than one value for the dimension. E.g. assuming the pool of bots have Ubuntu and Debian bots a dimension of "os": "Debian|Ubuntu" will resolve to use bots running either Debian or Ubuntu.
A dictionary with variables passed to gclient during a gclient sync operation. They are usually used to add or remove gclient dependencies.
"gclient_variables": { "download_android_deps": false }
The example above is used to avoid downloading the android sdk dependencies in builders that do not need it.
A list of strings representing flags passed to the tools/gn script. The strings can be in the form of “--flag=value” or “--flag” followed by “value”.
"gn": [ "--runtime-mode", "debug", "--prebuilt-dart-sdk", "--build-embedder-examples" ],
The previous example will prepare the configurations to build a host debug version using a prebuilt dart sdk and also build the embedder examples.
A dictionary with two keys: “config” which references the configs created by gn and “target” which is a list of strings with the Ninja targets to build.
"ninja": { "config": "host_debug", "targets": [ "flutter/build/archives:artifacts", "flutter/build/archives:embedder", ] },
In the example above the ninja command will use the configuration for host_debug and will build artifacts and embedder targets as described by the flutter/build/archives/BUILD.gn file.
This section of the build configuration is also known as local tests. It contains a list of dictionaries with configurations for scripts and parameters used to run tests inside the current build unit. These tests should not reference or use anything outside of the commit checkout or the outputs generated by running gn and ninja sections of the build configuration.
"tests": [ { "language": "python3", "name": "Host Tests for host_debug_impeller_vulkan", "parameters": [ "--variant", "host_debug_impeller_vulkan", "--type", "impeller-vulkan", "--engine-capture-core-dump" ], "script": "flutter/testing/run_tests.py" } ]
Description of the fields:
The test scripts will run in a deferred context (failing the step only after logs have been uploaded). Tester and builder recipes provide an environment variable called FLUTTER_LOGS_DIR pointing a temporary directory where the test runner can place any logs|artifacts needed to debug issues. At the end of the test execution the content of FLUTTER_LOGS_DIR will be uploaded to Google Cloud Storage before signaling the pass | fail test state.
Note that to keep the recipes generic they don’t know anything about what the test script is doing and it is the responsibility of the test script to copy the relevant files to the FLUTTER_LOGS_DIR directory.
Generators are scripts used to generate artifacts combining the output of two or more sub-builds. The most common use case is to generate universal binaries for Mac/iOS artifacts.
Generators can be written in any language but they are required to follow some guidelines to make them compatible with the engine build system.
The guidelines are as follows:
src/) directory. If there are global generators in a build configuration, the engine_v2 recipes will download the full sub-build archives to the src/out/<sub-build name> directory.Generators contain a single property “tasks” which is a list of tasks to be performed.
"generators": { "tasks": [] }
The example above represents a generator configuration with an empty list of tasks.
A task is a dictionary describing the scripts to be executed.
The property's description is as follows:
{ "name": "Debug-FlutterMacOS.framework", "parameters": [ "--dst", "out/debug", "--arm64-out-dir", "out/ios_debug", "--simulator-x64-out-dir", "out/ios_debug_sim", "--simulator-arm64-out-dir", "out/ios_debug_sim_arm64" ], "script": "flutter/sky/tools/create_full_ios_framework.py", "language": "python3" }
Global tests follow the same format as build tests but defined at the top level. The main difference from local tests is that global tests have access to the sub-build artifacts.
Global generators follow the same format as local generators but defined at the build top level. The main difference is that global generators can create new artifacts combining outputs of multiple sub-builds.
The archives component provides instructions to upload the artifacts generated by the global generators. Is a list of dictionaries with two keys: source and destination. source is a path relative to the checkout repository and destination is a relative path to <bucket>/flutter/<commit>.
"archives": [ { "source": "out/debug/artifacts.zip", "destination": "ios/artifacts.zip" }, { "source": "out/debug/ios-objcdoc.zip", "destination": "ios-objcdoc.zip" } ]
The example above will cause the file <checkout>/out/debug/artifacts.zip to be uploaded <bucket>/flutter/<commit>/ios/artifacts.zip.
Global generators can run locally if all their sub-build dependencies are downloaded. This section explains how to triage a local generator.
The instructions on this section can be used to triage problems with artifacts created by glocal generators(E.g.Debug|Release|Profile-ios-Flutter.xcframework) using the build outputs of CI subbuilds. During the migration to engine v2 we had a regression in the size of the flutter libraries, using this process we were able to inspect the files as they were generated by the CI, make changes to the generators and run the generators locally to validate the fixes.
CAS client is required to download the sub-build artifacts. To install it in your machine run the following steps:
mkdir $HOME/toolsCreate a gclient checkout following instructions from Setting up the engine environment.
CAS sub-build artifacts can be downloaded using information from a LUCI build. Using [https://ci.chromium.org/p/flutter/builders/prod/Mac%20mac_ios_engine/2818] as an example the execution details from steps 13 - 17 show the commands to download the archives. -dir parameter needs to be updated to point to the relative or full path to the out folder in your gclient checkout.
These are the commands to execute for our example build:
pushd <gclient checkout>/src/out cas download -cas-instance projects/chromium-swarm/instances/default_instance -digest 39f15436deaed30f861bdd507ba6297f2f26a2ff13d45acfd8819dbcda346faa/88 -dir ./ cas download -cas-instance projects/chromium-swarm/instances/default_instance -digest bdec3208e70ba5e50ee7bbedaaff4588d3f58167ad3d8b1c46d29c6ac3a18c00/94 -dir ./ cas download -cas-instance projects/chromium-swarm/instances/default_instance -digest d19edb65072aa9d872872b55d3c270db40c6a626c8a851ffcb457a28974f3621/84 -dir ./ cas download -cas-instance projects/chromium-swarm/instances/default_instance -digest ac6f08662d18502cfcd844771bae736f4354cb3fe209552fcf2181771e139e0b/86 -dir ./ cas download -cas-instance projects/chromium-swarm/instances/default_instance -digest 1d4d1a3b93847451fe69c1939d7582c0d728b198a40abd06f43d845117ef3214/86 -dir ./
The previous commands will create the ios_debug, ios_debug_sim, ios_debug_sim_arm64, ios_profile, and ios_release folders with the artifacts generated by its corresponding sub-builds.
Once the checkout and dependencies are available locally we can cd|pushd to the root of the client checkout and run the global generator. The command can be copied verbatim from the executions details of the build.
The following example will run the generator to create the ios artifacts:
python3 flutter/sky/tools/create_full_ios_framework.py --dst out/release \ --arm64-out-dir out/ios_release --simulator-x64-out-dir out/ios_debug_sim \ --simulator-arm64-out-dir out/ios_debug_sim_arm64 --dsym --strip