These are the steps we will/did follow to prune excessive history from the flutter/engine repository when we merged with flutter/flutter. The idea was to retain as much useful history as possible without blowing up the footprint of the framework's .git folder. The history that should get merged should be as relative and useful to currently engine development.
The engine .git folder is ~780MB of history.
Do not start with your working tree. Remove the origin so we don't mess with the flutter/engine.
############################################## ## Do some cleanup work on the engine and get ## the folder structure right. ############################################## # clone the repo to a fresh working folder git clone git@github.com:flutter/engine.git engine_prep cd engine_prep # for saftey - remove the remote - we're going to edit history git remote remove origin
If you want to analyze the repository, you should intall git filter-repo on your path and then run:
# Analyze if you want, just remember to remove .git/filter-repo git filter-repo --analyze --force
The output is stored in .git/filter-repo.
The following table is pulled from git-filter-repo's analsis. The Packed Size due to cross referencing. In general; we looked at large files that are not referenced any more and folders older than 2016.
| Packed Size | Deleted Date | Path | Notes |
|---|---|---|---|
| 112784745 | 2024-05-13 | ci/licenses_golden/licenses_third_party | |
| 27531902 | ~2021 | *.jar | binary |
| 27379931 | 2016-08-09 | third_party/android_platform | android_platform and webview |
| 27000000 | 2024-07-15 | impeller/docs/assets/*.(png | gif) |
| 15121375 | 2023-02-13 | *.ttc | font files |
| 10104182 | 2023-02-13 | /SourceHanSerifCN | |
| 7985682 | 2018-08-08 | travis | old ci |
| 6315637 | 2015-11-07 | examples/game | |
| 3939429 | 2015-07-28 | sky/sdk | |
| 3939429 | 2015-07-28 | sky/packages/sky | |
| 3903787 | 2016-08-09 | mojo | |
| 3686830 | 2022-06-14 | testing/scenario_app/android/reports | |
| 3188930 | 2015-06-30 | tests/fast | |
| 3173966 | 2015-08-07 | /example/game | |
| 2018961 | 2016-08-09 | third_party/libxml | |
| 1804199 | 2016-08-09 | third_party/tcmalloc | |
| 1393936 | ~2016 | *.dll | binary |
| 1373740 | 2017-07-06 | tests/data | |
| 1100665 | 2015-06-27 | benchmarks/parser/resources/html5.html | |
| 1059673 | 2015-07-20 | third_party/protobuf | |
| 978870 | 2022-04-27 | impeller/third_party | |
| 798852 | 2015-07-20 | third_party/cython | |
| 778560 | 2022-01-24 | lib/web_ui/test/golden_files | |
| 634455 | 2016-08-09 | third_party/libpng | |
| 610751 | 2024-05-13 | .golden | |
| 550475 | 2024-09-17 | impeller/fixtures/flutter_logo_baked.* | |
| 526837 | 2016-08-09 | third_party/libevent | |
| 523436 | 2015-07-20 | third_party/boringssl | |
| 514968 | 2022-04-27 | impeller/fixtures/image.png | |
| 461527 | 2015-12-11 | third_party/re2 | |
| 418122 | 2015-10-12 | examples/demo_launcher | |
| 413787 | 2015-11-07 | .aac | |
| 362787 | 2016-08-09 | third_party/glfw | |
| 349604 | 2016-08-09 | third_party/harfbuzz-ng | |
| 340869 | 2016-08-09 | third_party/okhttp | |
| 321659 | 2016-08-09 | .S | |
| 300824 | 2016-08-09 | .so | |
| 257633 | 2016-08-09 | third_party/libjpeg | |
| 257519 | 2016-08-09 | third_party/jinja2 | |
| 249618 | 2016-08-09 | third_party/zlib | |
| 218643 | 2015-12-11 | third_party/brotli | |
| 188622 | 2021-01-06 | .idl | |
| 184593 | 2015-09-02 | third_party/khronos | |
| 173210 | 2016-08-09 | .gypi | |
| 170484 | 2016-08-09 | third_party/expat | |
| 169578 | 2016-08-09 | .asm | |
| 161360 | 2016-08-09 | .m4 | |
| 142670 | 2018-05-10 | .in | |
| 140364 | 2015-12-11 | third_party/ots | |
| 137270 | 2016-08-09 | .hh | |
| 136787 | 2016-08-09 | .gyp | |
| 99503 | 2016-08-09 | third_party/qcms | |
| 91730 | 2015-08-21 | .pxd | |
| 84850 | 2016-08-09 | third_party/yasm |
The following command will remove files and foldes from the checkout history. Since this is a destructive edit, the SHA1 git hashes will be changed in the process. At the end, the .git history will be 74 MB of object files.
# Lets do some heavy filtering; # .git starts out at ~780MB and ends up at ~110MB git filter-repo --force --invert-paths \ --path-glob 'impeller/docs/assets/*.png' \ --path-glob 'impeller/docs/assets/*.gif' \ --path-glob '*/example/game/*' \ --path-glob 'benchmarks/parser/resources/html5.html' \ --path-glob '*.dll' \ --path-glob '*.jar' \ --path-glob '*/SourceHanSerifCN*' \ --path-glob 'third_party/txt/third_party/fonts/NotoSansCJK-Regular.ttc' \ --path-glob 'impeller/fixtures/flutter_logo_baked.*' \ --path-glob 'impeller/fixtures/image.png' \ --path-glob '*.golden' \ --path-glob '*.aac' \ --path-glob '*.S' \ --path-glob '*.so' \ --path-glob '*.idl' \ --path-glob '*.gpy' \ --path-glob '*.gypi' \ --path-glob '*.asm' \ --path-glob '*.m4' \ --path-glob '*.in' \ --path-glob '*.pxd' \ --path-glob '*.hh' \ --path 'ci/licenses_golden/licenses_third_party' \ --path 'testing/scenario_app/android/reports' \ --path 'impeller/third_party' \ --path 'mojo/public/third_party' \ --path 'tests/data' \ --path 'tests/fast' \ --path 'tests/framework' \ --path 'travis' \ --path 'mojo' \ --path 'sky/sdk' \ --path 'sky/engine' \ --path 'sky/tools/webkitpy' \ --path 'sky/shell' \ --path 'sky/packages/sky' \ --path 'sky/tests' \ --path 'sky/unit' \ --path 'sky/services' \ --path 'sky/compositor' \ --path 'sky/build' \ --path 'sky/specs' \ --path 'skysprites' \ --path 'examples/demo_launcher' \ --path 'examples/game' \ --path 'third_party/qcms' \ --path 'third_party/libevent' \ --path 'third_party/boringssl' \ --path 'third_party/tcmalloc' \ --path 'third_party/cython' \ --path 'third_party/protobuf' \ --path 'third_party/libpng' \ --path 'third_party/re2' \ --path 'third_party/harfbuzz-ng' \ --path 'third_party/jinja2' \ --path 'third_party/libjpeg' \ --path 'third_party/glfw' \ --path 'third_party/zlib' \ --path 'third_party/android_platform' \ --path 'third_party/expat' \ --path 'third_party/brotli' \ --path 'third_party/yasm' \ --path 'third_party/khronos' \ --path 'third_party/okhttp' \ --path 'third_party/libxml' \ --path 'third_party/ots' \ --path 'third_party/libXNVCtrl' \ --path 'lib/web_ui/test/golden_files' \ --path 'apk' \ --path 'flutter' \ --path 'base' \ --path 'sdk' \ --path 'gpu' \ --path 'engine' \ --path 'tools/webkitpy' \ --path 'tools/valgrind' \ --path 'tools/clang' \ --path 'tools/android' \ --path 'build/linux' \ --path 'build/win' \ --path 'build/mac' \ --path 'ui' \ --path 'examples/stocks' \ --path 'examples/stocks2' \ --path 'examples/stocks-fn' \ --path 'examples/data' \ --path 'examples/fitness' \ --path 'examples/city-list' \ --path 'examples/widgets' \ --path 'examples/raw' \ --path 'examples/color' \ --path 'examples/flights' \ --path 'examples/rendering' \ --path 'examples/fn' \ --path 'specs' \ --path 'url' \ --path 'services' \ --path 'framework' \ --path 'crypto' \ --path 'skia/ext' \ --path 'e2etests' \ --path 'tests/resources' \ --path 'viewer' \ --path 'lib/stub_ui' \ --path 'content_handler' # Garbage collect! git reflog expire --expire=now --all && git gc --prune=now --aggressive
The final destination for the engine source code will be in the directory engine/src/flutter except for DEPS which remains at the root. Using git mv only affects HEAD and can have some problems when logging. Instead we'll re-write history so it makes sense in the new world.
# Move files to engine/src/flutter, update tags so they don't collide, and move DEPS back to root. git filter-repo --to-subdirectory-filter engine/src/flutter --tag-rename '':'engine-' --force git filter-repo --path-rename engine/src/flutter/DEPS:DEPS
The PR link in the first line of the comment message will be wrong; flutter/flutter doesn't have the same history. To make history a little bit better, we only want to edit the first line. This must be done before we merge with the flutter/flutter repo so as not to step on their commit lines.
git filter-repo --force --message-callback ' return re.sub(br"^(.*)\((#\d+)\)\n(.*)", br"\1(flutter/engine\2)\n\3", message, 1) '
############################################## ## Now handle merging into flutter/flutter ############################################## git clone git@github.com:flutter/flutter.git flutter_merge cd flutter_merge # add the other tree as remote git remote add -f engine-upstream ~/src/engine_prep # --no-commit is important because we want to look around git merge --no-commit --allow-unrelated-histories engine-upstream/main # You're a wizard, Harry git commit -m "Merge flutter/engine into framework" # Garbage collect! # Now at 234MB .git git reflog expire --expire=now --all && git gc --prune=now --aggressive