Flutter Engine Build Definition Language uses json to describe a complex build using a combination of sub-builds, archives, generators and dependencies. It simplifies sharding by mapping build inputs to workflows listing explicitly the generated artifacts. The build definition language along with the Engine Recipes V2 and the generation of artifacts using GN+Ninja set the groundwork for efficient builds with dependency reusability.
Author: Godofredo Contreras (godofredoc)
Go Link: flutter.dev/go/engine-build-definition-language
Created: 01/2023 / **Last updated:**01/2023
Engine builds are a complex combination of recipe source code, gn+ninja commands, fuzzy dependencies hidden by running everything in a single builder, and multiple helper scripts using bash and python to generate artifacts combining build system outputs. This complexity is error prone and impacts the velocity of the whole team as adding|removing|updating artifacts require multiple coordinated pull requests and an in-depth knowledge of the engine source code, build system, recipes and infrastructure.
The build definition language targets the following goals:
Engine builds use LUCI recipes to build, test and archive artifacts. The workflow requires to checkout the engine repository, run gn, run ninja, create artifacts and upload them to GCS. These steps are implemented differently for different platforms usually by copy/pasting blocks of recipe code. This copy/paste process generated a large and complex engine.py recipe (2K+ lines of code) which is difficult to update/maintain without breaking the builds.
Having the build logic as a sequence of build steps in a monolithic recipe makes it very difficult to identify dependencies in between sub-builds, sometimes duplicating logic in multiple places and making it very difficult for engineers to add/remove logic.
Flutter Engine contributors that add/update/remove builds, platforms and artifacts from the build system.
To describe engine builds in a generic and scalable way we are creating a build definition language that supports all the current build use cases. The definition language is using json to describe builds, tests, archives, and dependency relations.
This definition language will be used by generic recipes to trigger independent build units in different machines using cas for intermediate storage and an orchestrator recipe that collects and integrates the multiple pieces.
On top of the current use cases the definition language also supports configurations for code signing and describing the artifacts explicitly. These two new use cases are required for the implementation of SLSA requirements and the implementation of a single command Flutter SDK release.
The definition language makes heavy use of GN and Ninja to move most of the build/test/archive logic to the flutter/engine repository. This will help us remove most of the logic from recipes and make the recipes implementation generic and reusable by Dart & Flutter teams.
Using a format different from json to describe the engine builds.
Engine builds will be translated one to one using the build definition language with one file per build. The build definition files will be stored in the flutter/engine/ci/builders directory.
The engine orchestrator recipe will read the file, shard builds, collect artifacts and upload them to the Google Cloud Storage bucket where they are downloaded by the Flutter Tool.
Ci_yaml file is the glue sticking all the components together. A new build will be added to the .ci.yaml file with a property pointing to the build definition file to be used by engine_v2 recipes. The following is an example of a build configuration referencing android_aot_engine.json:
- name: Mac mac_android_aot_engine
recipe: engine_v2/engine_v2
timeout: 60
properties:
config_name: mac_android_aot_engine
$flutter/osx_sdk : >-
{ "sdk_version": "14e222b" }
Flutter's LUCI Infrastructure uses the engine build configuration languageto build, test and archive the engine artifacts. The goal of the configurations using the build definition language is to move all the build, test and archive logic out of recipes and bring it to the engine repository.
The following is a non-exhaustive list of the benefits of moving to build configurations files:
To keep the build definition language simple the following assumptions were made during its design:
The build configuration is a json file containing a list of builds, tests, generators and archives. The following is an example of an empty configuration file:
{
"builds": [],
"tests": [],
"generators": {
"tasks": []
},
"archives": [
]
}
Build configuration files have to be checked into the engine/ci/builder directory where engine v2 recipes will be reading them from.
A configuration file defines a top level builder that will show up as a column in the Flutter Dashboard.
A build is a dictionary with a gn command, a ninja command, zero or more generator commands, zero or more local tests, zero or more local generators and zero or more output artifacts.
The following is the high level structure of the build component:
{
"archives": [],
"drone_dimensions": [],
"gclient_variables": {},
"gn": [],
"name": "host_debug",
"ninja": {},
"tests": []
}
Each build element will be translated to an independent sub-build and its entire out directory will be uploaded to CAS.
An archive component is used to tell the recipes which artifacts are generated by the build and where to upload them.
{
"name": "host_debug",
"base_path": "out/host_debug/zip_archives/",
"type": "gcs",
"include_paths": [
"out/host_debug/zip_archives/linux-x64/artifacts.zip"
]
}
Description of the fields:
A list of strings with key value pairs separated by an equal sign. These dimensions are used to select the bot where the sub-build will be running.
"drone_dimensions": [ "device_type=none", "os=Linux" ]
In the previous example, the build containing this drone_dimensions component will run on a bot with a Linux OS that does not have any devices attached to it.
A dictionary with the gclient variable as key and its content as value. This dictionary is passed to gclient during a gclient sync operation to add/remove gclient dependencies.
"gclient_variables": {
"download_android_deps": false
}
The example above is used to avoid downloading the android sdk dependencies in builders that do not need it.
A list of strings representing flags passed to the tools/gn script. The strings can be in the form of “--flag=value” or “--flag” followed by “value”.
"gn": [
"--runtime-mode",
"debug",
"--prebuilt-dart-sdk",
"--build-embedder-examples"
],
The previous example will prepare the configurations to build a host debug version using a prebuilt dart sdk and also build the embedder examples.
A dictionary with two keys: “config” which references the configs created by gn and “target” which is a list of strings with the Ninja targets to build.
"ninja": {
"config": "host_debug",
"targets": [
"flutter/build/archives:artifacts",
"flutter/build/archives:embedder",
]
},
In the example above the ninja command will use the configuration for host_debug and will build artifacts and embedder targets as described by the flutter/build/archives/BUILD.gn file.
This section of the build configuration will be referred to as the local tests. This section contains a list of dictionaries with configurations for the scripts and parameters used to run tests inside the current build unit. These tests should not reference or use anything outside of the commit checkout or the outputs generated by running the gn and ninja sections of the build config.
"tests": [
{
"language": "python3",
"name": "Host Tests for host_debug_impeller_vulkan",
"parameters": [
"--variant",
"host_debug_impeller_vulkan",
"--type",
"impeller-vulkan",
"--engine-capture-core-dump"
],
"script": "flutter/testing/run_tests.py",
"type": "local"
}
]
Description of the fields:
The test scripts will run in a deferred context (failing the step only after logs have been uploaded). The tester and builder recipes provide an environment variable called FLUTTER_LOGS_DIR pointing a temporary directory where the test runner can place any logs|artifacts needed to debug issues. At the end of the test execution the content of FLUTTER_LOGS_DIR will be uploaded to Google Cloud Storage before signaling the pass | fail test state.
Note that to keep the recipes generic they don’t know anything about what the test script is doing and it is the responsibility of the test script to copy the relevant files to the FLUTTER_LOGS_DIR directory.
Generators are scripts used to generate artifacts combining the output of two or more sub-builds. The most common use case is to generate FAT binaries for Mac/iOS artifacts.
Generators can be written in any language but they require to follow some guidance to make them compatible with the engine build system.
The guidelines are as follows:
Generators contain a single property “tasks” which is a list of tasks to be performed.
"generators": {
"tasks": []
}
The example above represents a generator configuration with an empty list of tasks.
Task is a dictionary describing the scripts to be executed.
The properties description is as follows:
Name, the name of the step running the script.
Parameters, flags passed to the script. Both input and output paths must be relative to the checkout directory.
Script, the script path relative to the checkout repository.
Language, the script language executable to run the script. If empty it is assumed to be bash.
{
"name": "Debug-FlutterMacOS.framework",
"parameters": [
"--dst",
"out/debug",
"--arm64-out-dir",
"out/ios_debug",
"--simulator-x64-out-dir",
"out/ios_debug_sim",
"--simulator-arm64-out-dir",
"out/ios_debug_sim_arm64"
],
"script": "flutter/sky/tools/create_full_ios_framework.py",
"language": "python3"
}
The archives component provides instructions to upload the artifacts generated by the global generators. Is a list of dictionaries with two keys: source and destination. Source is a path relative to the checkout repository and destination is a relative path to <bucket>/flutter/<commit>.
"archives": [ { "source": "out/debug/artifacts.zip", "destination": "ios/artifacts.zip" }, { "source": "out/debug/ios-objcdoc.zip", "destination": "ios-objcdoc.zip" } ]
The example above will cause the file <checkout>/out/debug/artifacts.zip to be uploaded <bucket>/flutter/<commit>/ios/artifacts.zip.
N/A
N/A.
The build definition language will force all the artifacts to be generated with GN+Ninja. This will require a carefully designed strategy to replace the artifacts with minimal disruption.
A migration to simpler and more scalable recipes will be required to achieve all the goals of the build definition language. A deprecation of the old recipes will be also necessary to remove all the technical debt.
From the developers point of view we can expect some disruption with multiple land -> revert -> re-land cycles. For Flutter users there will be no difference as they will receive the same files with the same content even though they are generated differently.
Configuration files will be created and used with builders running in parallel to the production ones. The builders using the new build configuration files will run on the staging environment for several weeks before moving them to production.
The artifacts generated by GN+Ninja will be released with the old recipes to ensure their content is correct and complete.
Artifacts generated by builders using the configuration language will be using GN+Ninja artifacts exclusively. Once all those artifacts are validated with the old recipes the builds using the configuration language will be moved to production and the old recipes deprecated and removed.
The main purpose of this document is to describe the build definition language. Once it has been reviewed most of the content of this document will be adapted to markup and will be merged along the build configuration files in the engine repository.
This document, the document describing GN+Ninja artifacts, the document describing the engine v2 recipes and the document with the plan to speed up engine builds will be added to the flutter wiki page.
The migration to use the build definition language depends on the release | validation of the GN + Ninja artifacts. Once that part is complete two changes will need to be coordinated:
Global generators can run locally if all their sub-build dependencies are downloaded. This section explains how to triage a local generator.
CAS client is required to download the sub-build artifacts. To install it in your machine run the following steps:
mkdir $HOME/toolsCreate a gclient checkout following instructions from Setting up the engine environment.
CAS sub-build artifacts can be downloaded using information from a LUCI build. Using https://ci.chromium.org/p/flutter/builders/prod/Mac%20mac_ios_engine/2818 as an example the execution details from steps 13 - 17 show the commands to download the archives. -dir parameter needs to be updated to point to the relative or full path to the out folder in your gclient checkout.
These are the commands to execute for our example build:
pushd <gclient checkout>/src/out cas download -cas-instance projects/chromium-swarm/instances/default_instance -digest 39f15436deaed30f861bdd507ba6297f2f26a2ff13d45acfd8819dbcda346faa/88 -dir ./ cas download -cas-instance projects/chromium-swarm/instances/default_instance -digest bdec3208e70ba5e50ee7bbedaaff4588d3f58167ad3d8b1c46d29c6ac3a18c00/94 -dir ./ cas download -cas-instance projects/chromium-swarm/instances/default_instance -digest d19edb65072aa9d872872b55d3c270db40c6a626c8a851ffcb457a28974f3621/84 -dir ./ cas download -cas-instance projects/chromium-swarm/instances/default_instance -digest ac6f08662d18502cfcd844771bae736f4354cb3fe209552fcf2181771e139e0b/86 -dir ./ cas download -cas-instance projects/chromium-swarm/instances/default_instance -digest 1d4d1a3b93847451fe69c1939d7582c0d728b198a40abd06f43d845117ef3214/86 -dir ./
The previous commands will create the ios_debug, ios_debug_sim, ios_debug_sim_arm64, ios_profile, and ios_release folders with the artifacts generated by its corresponding sub-builds.
Once the checkout and dependencies are available locally we can cd|pushd to the root of the client checkout and run the global generator. The command can be copied verbatim from the executions details of the build.
The following example will run the generator to create the ios artifacts:
python3 flutter/sky/tools/create_full_ios_framework.py --dst out/release \ --arm64-out-dir out/ios_release --simulator-x64-out-dir out/ios_debug_sim \ --simulator-arm64-out-dir out/ios_debug_sim_arm64 --dsym --strip