Engine V2 recipes were conceived as a way to speed up engine builds by sharding them into multiple sub-builds. That initial idea has evolved into a generic solution that speeds up builds, improve reusability, simplify the build workflows, improve presubmit testing of new artifacts and in general improves visibility of what the build system is generating.
Author: Godofredo Contreras (godofredoc) Created: 02/2023 / Last updated: 08/2023
Recipes is a domain specific language to describe sequences of steps in a cross-platform and testable way. The chromium team has created multiple modules that simplify the communication with LUCI which are the services used by Flutter to build and test.
There are a few reasons why the use of recipes is very complicated in the Flutter Infrastructure but the most important one is that we are putting too much business logic in a place designed to orchestrate tasks in multiple platforms emphasizing scalability over simplicity.
This document outlines the design of generic recipes that will rarely change. It will explain in detail how the business logic will be moved to the engine repository and how these recipes will be used as a foundational block to implement build dependency graphs and optimize build and test executions.
Flutter contributors adding new features to Engine V2 recipes and improving the Build Configuration Language.
Engine V2 recipes implementation uses three recipes: engine_v2/engine_v2.py (orchestrator),engine_v2/builder.py (builder), engine_v2/tester.py (tester) and several recipe modules: shard_util_v2, archives, and flutter_deps.
The recipes and modules are highly customizable through a build configuration file making it possible to create a completely new build without modifying a single line of code in the recipes repository.
Engine V2 recipes are designed to automatically shard sub-builds with the possibility of mixing and matching platforms. It is possible to build an artifact in a Linux machine (e.g. Javascript tests) and use it in sub-builds targeting different platforms. Furthermore the orchestrator is designed as an intermediate step to support dependency graphs, reusability of builds and to make efficient use of resources.
Code signing is not only getting automated but it is also getting a major revamp to simplify the entire process separating responsibilities (build/test/archive separation).
Security is another major benefit of Engine V2 recipes allowing release candidate branch artifacts to be built in SLSA compliant infrastructure reusing the same recipes and build configurations.
Engine V2 recipes are opening the possibility to share infrastructure between Dart and Flutter with minimal effort setting the groundwork for building a logical monorepo where changes in dart, engine, flutter, packages, etc can be built and tested as a single unit.
Engine V2 recipes will be used in conjunction with the Engine Build Definition Language and GN+Ninja Artifacts to optimize, simplify and speed up the Flutter Engine Builds.
A build configuration file will be created in flutter/engine describing a build, a target referencing the configuration will be added to .ci.yaml. The Cocoon Backend will use them to trigger builds in LUCI infrastructure.
A second use case will reuse the build configuration files and .ci.yaml to trigger builds in SLSA compliant builders using a recipes orchestrator.
Build configuration files describe the build components. Detailed information can be found here. These configurations are used by engine_v2.py to shard builds, wait for their completion and run any post-build tasks that require inputs from multiple sub-builds.
.ci.yaml is a per repository file used to instruct the LUCI infrastructure what to run, which environments to use, and the platform configurations to run the builds on. Documentation for .ci.yaml can be found here. For the purpose of this document the engine .ci.yaml file will be used.
Engine V2 build configuration is very simple, the following is an example:
- name: Mac mac_android_aot_engine recipe: engine_v2/engine_v2 timeout: 60 properties: config_name: mac_android_aot_engine $flutter/osx_sdk : >- { "sdk_version": "14a5294e" }
Where name is the title in the task in the flutter-dashboard, recipe is the engine v2 orchestrator recipe, timeout represents for how many minutes the build is allowed to run before timing out and properties is a list of key/value pairs with additional information to be passed to the build.
This is a group of recipes collaborating to run builds/tests efficiently using the available resources. This group currently includes: engine_v2(orchestrator), builder, tester and signer.
The following are the design principles for these recipes:
Entry point recipe which reads the build configuration file, triggers multiple sub-builds, waits for builds to complete, runs global generators and signs the generated artifacts.
There is room for optimization like running global generators as soon as their dependencies are built, using low resource machines as orchestrators. These optimizations were intentionally left out as a build graph scheduler will be implemented as soon as the migration to engine build v2 is completed.
This is a recipe that knows how to run GN|Ninja commands and how to run scripts in different languages.
It receives a build configuration with instructions of which gn and ninja commands to run and the tests to run on the outputs of build.
This recipe knows how to run scripts(tests) using the outputs of a sub-build. An usage example of this functionality is Web Engine builds Javascript tests and then triggers several tester sub-builds in different platforms.
This may not seem like a huge improvement over the status quo but it really is. It removes all the overhead of setting up the build environment, e.g. third_party dependencies, xcode, android sdk, etc.
Using the tester sub-builds efficiently requires foundational changes where the build system prepares everything required to run the tests in a way that the tester only needs to know how to download the sub-build artifacts, run it and provide a pass|fail signal. In other words, global tests become self contained artifacts that include everything needed to run.
Note: Requiring checkouts of the source code, files in certain locations etc. removes most of the benefits of separating build/tests/archives.
Builds and global tests are automatically sharded by the orchestrator, injecting dependencies described in the build configuration file to builder|tester tasks.
Dart & Flutter have been investing heavily in SLSA compliant workflows. SLSA requires secure infrastructure where human intervention is minimal with multi party approval.
Engine v2 builds provide the foundational work required to run the builds in different environments using the same configurations, additional logging, audit trails and multi-party approval. All of this without impacting the velocity of the development workflows.
Release builder is implemented with <50 lines of code. It takes .ci.yaml and the build configurations to produce SLSA compliant artifacts for release candidate branches.
There is functionality that can be reused across multiple recipes. One example is engine_v2 and release_builder recipes triggering sub-builds or code to set up the engine build environment. One of the reasons we were able to implement release_builder with minimal lines of code is because it is reusing functionality already implemented. Structuring functionality as recipe_modules help us to implement fixes in a single place and apply to all the recipes. Although there are many modules, only the three most important ones are covered in this document: shard_util_v2, archives, and display_util.
This module provides the core functionality supporting Engine V2 Builds. It understands the build configuration language files and helps trigger multiple sub-builds based on those configurations. It also supports automated executions of builds using BuildBucket or manual execution using Led.
This module provides generic functionality to archive artifacts using different environments.
This module provides functionality to display builds in a human readable format. It encapsulates all the functionality to display sub-builds in a single place. This simplifies improving user interface changes and rolling it to all the flutter infrastructure at once.
The recipe modules will be reused in both legacy and engine v2 recipes and the builds using the two versions of recipes will coexist for some time.
All the work described in this document will integrate with no additional effort with .ci.yaml and with the cocoon scheduler for the development workflows.