EXPERIMENT: evaluating compression vs interning on Wakelock

NOT FOR REVIEW

Prompt
We are going to do some investigation about interning vs compression.
You'll have to write some python or C++ throwaway scripts to help me with some research on the topic.

Pre-work:
Take the long_trace.pftrace file and extract out only the TracePacket that contain an AppWakelockBundle (defined in app_wakelock_data.proto). Save that in wakelock.pftrace.

Context:
you'll notice that wakelocks are stored in an efficient manner via a union of interning + columnar encoding
- There is an AppWakelockInfo structure which is interned.
- When a wakelock is detected and stored into the trace:
 - If the wakelock is new, a new AppWakelockInfo is created and appended to TracePacket.interned_data.app_wakelock_info (see interned_data.proto)
 - The index of the newly created intern entry is appended to intern_id
 - The timestamp of the even is appended to encoded_ts
 - Ignore the other two fiels AppWakelockInfo info = 3; and acquired = 4

So the way you interpret the data is that encoded_ts and intern_id have the same cardinality. intern_id refers to the entry previously emitted in interned_data

Assignment:
I want you to de-intern the trace, and pretend that the AppWakelockInfo was written as a record, in a dumb way, without any interning.
SO I want you to do this:

- Modify TrackEvent and add AppWakelockInfo as one of the possible fields
- Write a python/c++ script that parses the input trace and de-interns the data. FOr each entry in encoded_ts + intern_id, you shall create a new TrackEvent of type instant, with the appopriate timestamp, and same AppWakelockInfo fields (skip the iid field because it's only used for interning)
- Hence create a wakelock_uninterned.pftrace, taking the previous wakelock.pftrace and expanding each wakelock as described above
- Then I want you to update the importer code in trace_processor to import also this file. I want you to verify that you did a good job ensuring that at the trace_processor level, the data looks identical. You'll have to join the slice table with the tracks table, and filter for track of type 'app_wakelock_events', and then check the slice and its args

After you have done all this, I want you to take both traces (wakelock.pftrace and wakelock_uninterned.pftrace) and apply compression as follows. You have to try the combination of the following matrix
zstd levels={10, 15, 19} with block sizes of 512K, 1MB, 2MB, 8MB (that is, apply compression indpendently to blocks of these sizes)

then write me a report with comparing the reults of this compression matrix on both traces in a tabular format
5 files changed
tree: 11ee3ff91d69010e4584a658d319ba1f549b8d3b
  1. .github/
  2. bazel/
  3. build_overrides/
  4. buildtools/
  5. contrib/
  6. docs/
  7. examples/
  8. gn/
  9. include/
  10. infra/
  11. protos/
  12. python/
  13. src/
  14. test/
  15. third_party/
  16. tools/
  17. ui/
  18. .bazelignore
  19. .bazelrc
  20. .bazelversion
  21. .clang-format
  22. .clang-tidy
  23. .git-blame-ignore-revs
  24. .gitallowed
  25. .gitattributes
  26. .gitignore
  27. .gn
  28. .style.yapf
  29. Android.bp
  30. Android.bp.extras
  31. BUILD
  32. BUILD.extras
  33. BUILD.gn
  34. CHANGELOG
  35. CONTRIBUTORS.txt
  36. DIR_METADATA
  37. heapprofd.rc
  38. LICENSE
  39. meson.build
  40. METADATA
  41. MODULE.bazel
  42. MODULE.bazel.lock
  43. MODULE_LICENSE_APACHE2
  44. OWNERS
  45. OWNERS.github
  46. perfetto.rc
  47. perfetto_flags.aconfig
  48. PerfettoIntegrationTests.xml
  49. persistent_cfg.pbtxt
  50. README.chromium
  51. README.md
  52. TEST_MAPPING
  53. traced_perf.rc
  54. WORKSPACE
README.md

Perfetto - System profiling, app tracing and trace analysis

Perfetto is an open-source suite of SDKs, daemons and tools which use tracing to help developers understand the behaviour of complex systems and root-cause functional and performance issues on client and embedded systems.

It is a production-grade tool that is the default tracing system for the Android operating system and the Chromium browser.

Core Components

Perfetto is not a single tool, but a collection of components that work together:

  • High-performance tracing daemons: For capturing tracing information from many processes on a single machine into a unified trace file.
  • Low-overhead tracing SDK: A C++17 library for direct userspace-to-userspace tracing of timings and state changes in your application.
  • Extensive OS-level probes: For capturing system-wide context on Android and Linux (e.g. scheduling states, CPU frequencies, memory profiling, callstack sampling).
  • Browser-based UI: A powerful, fully local UI for visualizing and exploring large, multi-GB traces on a timeline. It works in all major browsers, requires no installation, and can open traces from other tools.
  • SQL-based analysis library: A powerful engine that allows you to programmatically query traces using SQL to automate analysis and extract custom metrics.

Why Use Perfetto?

Perfetto was designed to be a versatile and powerful tracing system for a wide range of use cases.

  • For Android App & Platform Developers: Debug and root-cause functional and performance issues like slow startups, dropped frames (jank), animation glitches, low memory kills, and ANRs. Profile both Java/Kotlin and native C++ memory usage with heap dumps and profiles.
  • For C/C++ Developers (Linux, macOS, Windows): Use the Tracing SDK to instrument your application with custom trace points to understand its execution flow, find performance bottlenecks, and debug complex behavior. On Linux, you can also perform detailed CPU and native heap profiling.
  • For Linux Kernel & System Developers: Get deep insights into kernel behavior. Perfetto acts as an efficient userspace daemon for ftrace, allowing you to visualize scheduling, syscalls, interrupts, and custom kernel tracepoints on a timeline.
  • For Chromium Developers: Perfetto is the tracing backend for chrome://tracing. Use it to debug and root-cause issues in the browser, V8, and Blink.
  • For Performance Engineers & SREs: Analyze and visualize a wide range of profiling and tracing formats, not just Perfetto's. Use the powerful SQL interface to programmatically analyze traces from tools like Linux perf, macOS Instruments, Chrome JSON traces, and more.

Getting Started

We‘ve designed our documentation to guide you to the right information as quickly as possible, whether you’re a newcomer to performance analysis or an experienced developer.

  1. New to tracing? If you're unfamiliar with concepts like tracing and profiling, start here:

  2. Ready to dive in? Our “Getting Started” guide is the main entry point for all users. It will help you find the right tutorials and documentation for your specific needs:

  3. Want the full overview? For a comprehensive look at what Perfetto is, why it's useful, and who uses it, see our main documentation page:

Debian Distribution

For users interested in the Debian distribution of Perfetto, the official source of truth and packaging efforts are maintained at Debian Perfetto Salsa Repository

Community & Support

Have questions? Need help?

We follow Google's Open Source Community Guidelines.