docs/contributing/testing/How-to-write-a-memory-test-for-Flutter.md - mirrors/flutter - Git at Google

 There are many ways to write a memory test for Flutter. In this article, we give 3 classes of example tests that are currently used by Flutter device lab. Memory performance is a high priority for Flutter so there are many new memory tools and test utilities in progress. We’ll add them in this doc in the future.

 ## [MemoryTest][class MemoryTest] that interacts with adb directly

 These memory tests use the [MemoryTest class defined in the device lab perf_tests.dart][class MemoryTest] to poll adb directly before and after an overridable `useMemory` function. By default, `useMemory` will just run an app in release and wait for a “done” message to be printed in logcat.

 Examples include
 - [`complex_layout_scroll_perf__memory`][complex layout memory manifest]
   - [device lab task file][complex layout memory task]
   - [main file][complex layout memory main]
 - [`fast_scroll_large_images_memory`][fast scroll memory manifest]
   - [device lab task file][fast scroll memory task]
   - [main file][fast scroll memory main]

 To write a new MemoryTest case `some_memory_perf` and add it to Flutter’s device lab so Flutter’s CI system can measure it for each Flutter commit, follow examples above to

 1. Create a `main` function for the test app in a file named like `test_memory/some_memory_perf.dart`.
 2. Add a `some_memory_perf` entry to [manifest.yaml][manifest]
 3. Add a `some_memory_perf.dart` file to [dev/devicelab/bin/tasks][tasks] folder.


 ### Pros
 - Low overhead.
 - Works in all runtime modes, including release.
 - The test has complete control on when to start and stop the memory measurement.

 ### Cons
 - Only have 2 memory readings, begin and end, during the app run.
 - Polling ADB may trigger collections of the Java heap.
 - Only works on Android targets.
 - Requires a test environment with access to ADB.
 - Requires a host machine with Flutter SDK installed.

 ## DevTools Memory Test

 The memory tests use DevTools to poll adb and Dart VM during a normal Flutter driver test run, which typically measures speed performance instead of memory performance. [DevToolsMemoryTest][class DevToolsMemoryTest] handles most of the process so a new test only needs to specify the driver test location.

 Examples include
 - [`complex_layout_scroll_perf__devtools_memory`][complex layout devtools memory manifest]
   - [device lab task file][complex layout devtools memory task]
 - [`large_image_changer_perf_android`][large image changer manifest]
   - [device lab task file][large image changer task]

 To write a new DevTools memory test case `some_memory_perf` and add it to Flutter’s device lab so Flutter’s CI system can measure it for each Flutter commit, follow examples above to

 1. Write (or reuse) a normal Flutter driver test for the app in files named like `test_driver/some_memory_perf.dart` and `test_driver/some_memory_perf_test.dart`.
 2. Add a `some_memory_perf` entry to [manifest.yaml][manifest]
 3. Add a `some_memory_perf.dart` file to [dev/devicelab/bin/tasks][tasks] folder.

 ### Pros
 - Have finer grained measurements (~1 reading per second).
 - Also have Dart VM memory info.
 - Can easily turn a speed-focused driver test into a memory test.

 ### Cons
 - Don’t have much control on when to start and stop the measurement.
 - Polling ADB may trigger collections on the Java heap.
 - Requires a test environment with access to ADB.
 - Only works on Android targets.
 - Not available for release mode, so may incur extra memory overhead in profile or debug mode.
 - Requires a host machine with Flutter SDK installed.

 ## iOS Memory Test

 The iOS embedding of Flutter supports sampling memory usage during runtime, which then writes metrics to the [Dart timeline][Dart timeline]. After recording a timeline for the relevant portion of an application’s execution, the timeline can be analyzed to obtain memory related information from the profile.

 Examples include
 - [`large_image_changer_perf_ios`][large image changer manifest ios]
   - [device lab task file][large image changer task ios]

 To write a new iOS memory test case `some_memory_perf` and add it to Flutter’s device lab so Flutter’s CI system can measure it for each Flutter commit, follow examples above to

 1. Write (or reuse) a normal Flutter driver test for the app in files named like `test_driver/some_memory_perf.dart` and `test_driver/some_memory_perf_test.dart`.
 2. Add a `some_memory_perf` entry to [manifest.yaml][manifest]
 3. Add a `some_memory_perf.dart` file to [dev/devicelab/bin/tasks][tasks] folder that specifies `measureMemory: true`.

 ### Pros
 - Can be run on a machine that does not have the Flutter SDK installed.
 - Can adjust the sampling frequency so one can have as many or as few measurements as needed.
 - Each sampling has much less overhead compared to calling adb
 - Flutter driver tests on iOS get memory measurements for free.

 ### Cons
 - Only works on iOS targets.
 - Not available for release mode, so may incur extra memory overhead in profile or debug mode.
 - Memory polling mechanism may incur additional memory overhead.


 [manifest]: https://github.com/flutter/flutter/blob/main/dev/devicelab/manifest.yaml

 [tasks]: https://github.com/flutter/flutter/tree/main/dev/devicelab/bin/tasks

 [class MemoryTest]: https://github.com/flutter/flutter/blob/51bb11f7cece47840a9ee6d6d43db97ab16b31df/dev/devicelab/lib/tasks/perf_tests.dart#L941

 [complex layout memory manifest]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/manifest.yaml#L329

 [complex layout memory task]: https://github.com/flutter/flutter/blob/main/dev/devicelab/bin/tasks/complex_layout_scroll_perf__memory.dart

 [complex layout memory main]: https://github.com/flutter/flutter/blob/main/dev/benchmarks/complex_layout/test_memory/scroll_perf.dart

 [fast scroll memory manifest]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/manifest.yaml#L837

 [fast scroll memory task]: https://github.com/flutter/flutter/blob/main/dev/devicelab/bin/tasks/fast_scroll_large_images__memory.dart

 [fast scroll memory main]: https://github.com/flutter/flutter/blob/main/dev/benchmarks/macrobenchmarks/test_memory/large_images.dart

 [class DevToolsMemoryTest]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/lib/tasks/perf_tests.dart#L1138

 [complex layout devtools memory manifest]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/manifest.yaml#L359

 [complex layout devtools memory task]: https://github.com/flutter/flutter/blob/main/dev/devicelab/bin/tasks/complex_layout_scroll_perf__devtools_memory.dart

 [large image changer manifest]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/manifest.yaml#L874

 [large image changer task]: https://github.com/flutter/flutter/blob/main/dev/devicelab/bin/tasks/large_image_changer_perf_android.dart

 [Dart timeline]:https://flutter.dev/docs/development/tools/devtools/timeline

 [large image changer manifest ios]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/manifest.yaml#L880

 [large image changer task ios]: https://github.com/flutter/flutter/blob/main/dev/devicelab/bin/tasks/large_image_changer_perf_ios.dart
	There are many ways to write a memory test for Flutter. In this article, we give 3 classes of example tests that are currently used by Flutter device lab. Memory performance is a high priority for Flutter so there are many new memory tools and test utilities in progress. We’ll add them in this doc in the future.

	## [MemoryTest][class MemoryTest] that interacts with adb directly

	These memory tests use the [MemoryTest class defined in the device lab perf_tests.dart][class MemoryTest] to poll adb directly before and after an overridable `useMemory` function. By default, `useMemory` will just run an app in release and wait for a “done” message to be printed in logcat.

	Examples include
	- [`complex_layout_scroll_perf__memory`][complex layout memory manifest]
	- [device lab task file][complex layout memory task]
	- [main file][complex layout memory main]
	- [`fast_scroll_large_images_memory`][fast scroll memory manifest]
	- [device lab task file][fast scroll memory task]
	- [main file][fast scroll memory main]

	To write a new MemoryTest case `some_memory_perf` and add it to Flutter’s device lab so Flutter’s CI system can measure it for each Flutter commit, follow examples above to

	1. Create a `main` function for the test app in a file named like `test_memory/some_memory_perf.dart`.
	2. Add a `some_memory_perf` entry to [manifest.yaml][manifest]
	3. Add a `some_memory_perf.dart` file to [dev/devicelab/bin/tasks][tasks] folder.


	### Pros
	- Low overhead.
	- Works in all runtime modes, including release.
	- The test has complete control on when to start and stop the memory measurement.

	### Cons
	- Only have 2 memory readings, begin and end, during the app run.
	- Polling ADB may trigger collections of the Java heap.
	- Only works on Android targets.
	- Requires a test environment with access to ADB.
	- Requires a host machine with Flutter SDK installed.

	## DevTools Memory Test

	The memory tests use DevTools to poll adb and Dart VM during a normal Flutter driver test run, which typically measures speed performance instead of memory performance. [DevToolsMemoryTest][class DevToolsMemoryTest] handles most of the process so a new test only needs to specify the driver test location.

	Examples include
	- [`complex_layout_scroll_perf__devtools_memory`][complex layout devtools memory manifest]
	- [device lab task file][complex layout devtools memory task]
	- [`large_image_changer_perf_android`][large image changer manifest]
	- [device lab task file][large image changer task]

	To write a new DevTools memory test case `some_memory_perf` and add it to Flutter’s device lab so Flutter’s CI system can measure it for each Flutter commit, follow examples above to

	1. Write (or reuse) a normal Flutter driver test for the app in files named like `test_driver/some_memory_perf.dart` and `test_driver/some_memory_perf_test.dart`.
	2. Add a `some_memory_perf` entry to [manifest.yaml][manifest]
	3. Add a `some_memory_perf.dart` file to [dev/devicelab/bin/tasks][tasks] folder.

	### Pros
	- Have finer grained measurements (~1 reading per second).
	- Also have Dart VM memory info.
	- Can easily turn a speed-focused driver test into a memory test.

	### Cons
	- Don’t have much control on when to start and stop the measurement.
	- Polling ADB may trigger collections on the Java heap.
	- Requires a test environment with access to ADB.
	- Only works on Android targets.
	- Not available for release mode, so may incur extra memory overhead in profile or debug mode.
	- Requires a host machine with Flutter SDK installed.

	## iOS Memory Test

	The iOS embedding of Flutter supports sampling memory usage during runtime, which then writes metrics to the [Dart timeline][Dart timeline]. After recording a timeline for the relevant portion of an application’s execution, the timeline can be analyzed to obtain memory related information from the profile.

	Examples include
	- [`large_image_changer_perf_ios`][large image changer manifest ios]
	- [device lab task file][large image changer task ios]

	To write a new iOS memory test case `some_memory_perf` and add it to Flutter’s device lab so Flutter’s CI system can measure it for each Flutter commit, follow examples above to

	1. Write (or reuse) a normal Flutter driver test for the app in files named like `test_driver/some_memory_perf.dart` and `test_driver/some_memory_perf_test.dart`.
	2. Add a `some_memory_perf` entry to [manifest.yaml][manifest]
	3. Add a `some_memory_perf.dart` file to [dev/devicelab/bin/tasks][tasks] folder that specifies `measureMemory: true`.

	### Pros
	- Can be run on a machine that does not have the Flutter SDK installed.
	- Can adjust the sampling frequency so one can have as many or as few measurements as needed.
	- Each sampling has much less overhead compared to calling adb
	- Flutter driver tests on iOS get memory measurements for free.

	### Cons
	- Only works on iOS targets.
	- Not available for release mode, so may incur extra memory overhead in profile or debug mode.
	- Memory polling mechanism may incur additional memory overhead.


	[manifest]: https://github.com/flutter/flutter/blob/main/dev/devicelab/manifest.yaml

	[tasks]: https://github.com/flutter/flutter/tree/main/dev/devicelab/bin/tasks

	[class MemoryTest]: https://github.com/flutter/flutter/blob/51bb11f7cece47840a9ee6d6d43db97ab16b31df/dev/devicelab/lib/tasks/perf_tests.dart#L941

	[complex layout memory manifest]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/manifest.yaml#L329

	[complex layout memory task]: https://github.com/flutter/flutter/blob/main/dev/devicelab/bin/tasks/complex_layout_scroll_perf__memory.dart

	[complex layout memory main]: https://github.com/flutter/flutter/blob/main/dev/benchmarks/complex_layout/test_memory/scroll_perf.dart

	[fast scroll memory manifest]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/manifest.yaml#L837

	[fast scroll memory task]: https://github.com/flutter/flutter/blob/main/dev/devicelab/bin/tasks/fast_scroll_large_images__memory.dart

	[fast scroll memory main]: https://github.com/flutter/flutter/blob/main/dev/benchmarks/macrobenchmarks/test_memory/large_images.dart

	[class DevToolsMemoryTest]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/lib/tasks/perf_tests.dart#L1138

	[complex layout devtools memory manifest]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/manifest.yaml#L359

	[complex layout devtools memory task]: https://github.com/flutter/flutter/blob/main/dev/devicelab/bin/tasks/complex_layout_scroll_perf__devtools_memory.dart

	[large image changer manifest]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/manifest.yaml#L874

	[large image changer task]: https://github.com/flutter/flutter/blob/main/dev/devicelab/bin/tasks/large_image_changer_perf_android.dart

	[Dart timeline]:https://flutter.dev/docs/development/tools/devtools/timeline

	[large image changer manifest ios]: https://github.com/flutter/flutter/blob/7e41425d4af21dec7a7ff072a3ec1387859e32c8/dev/devicelab/manifest.yaml#L880

	[large image changer task ios]: https://github.com/flutter/flutter/blob/main/dev/devicelab/bin/tasks/large_image_changer_perf_ios.dart