| # Trace configuration |
| |
| Unlike many always-on logging systems (e.g. Linux's rsyslog, Android's logcat), |
| in Perfetto all tracing data sources are idle by default and record data only |
| when instructed to do so. |
| |
| Data sources record data only when one (or more) tracing sessions are active. |
| A tracing session is started by invoking the `perfetto` cmdline client and |
| passing a config (see QuickStart guide for |
| [Android](/docs/quickstart/android-tracing.md), |
| [Linux](/docs/quickstart/linux-tracing.md), or [Chrome on desktop](/docs/quickstart/chrome-tracing.md)). |
| |
| A simple trace config looks like this: |
| |
| ```protobuf |
| duration_ms: 10000 |
| |
| buffers { |
| size_kb: 65536 |
| fill_policy: RING_BUFFER |
| } |
| |
| data_sources { |
| config { |
| name: "linux.ftrace" |
| target_buffer: 0 |
| ftrace_config { |
| ftrace_events: "sched_switch" |
| ftrace_events: "sched_wakeup" |
| } |
| } |
| } |
| |
| ```` |
| |
| And is used as follows: |
| |
| ```bash |
| perfetto --txt -c config.pbtx -o trace_file.perfetto-trace |
| ``` |
| |
| TIP: Some more complete examples of trace configs can be found in the repo in |
| [`/test/configs/`](/test/configs/). |
| |
| NOTE: If you are tracing on Android using adb and experiencing problems, see |
| [the Android section](#android) below. |
| |
| ## TraceConfig |
| |
| The TraceConfig is a protobuf message |
| ([reference docs](/docs/reference/trace-config-proto.autogen)) that defines: |
| |
| 1. The general behavior of the whole tracing system, e.g.: |
| * The max duration of the trace. |
| * The number of in-memory buffers and their size. |
| * The max size of the output trace file. |
| |
| 2. Which data sources to enable and their configuration, e.g.: |
| * For the [kernel tracing data source](/docs/data-sources/cpu-scheduling.md) |
| , which ftrace events to enable. |
| * For the [heap profiler](/docs/data-sources/native-heap-profiler.md), the |
| target process name and sampling rate. |
| |
| See the _data sources_ section of the docs for details on how to |
| configure the data sources bundled with Perfetto. |
| |
| 3. The `{data source} x {buffer}` mappings: which buffer each data |
| source should write into (see [buffers section](#buffers) below). |
| |
| The tracing service (`traced`) acts as a configuration dispatcher: it receives |
| a config from the `perfetto` cmdline client (or any other |
| [Consumer](/docs/concepts/service-model.md#consumer)) and forwards parts of the |
| config to the various [Producers](/docs/concepts/service-model.md#producer) |
| connected. |
| |
| When a tracing session is started by a consumer, the tracing service will: |
| |
| * Read the outer section of the TraceConfig (e.g. `duration_ms`, `buffers`) and |
| use that to determine its own behavior. |
| * Read the list of data sources in the `data_sources` section. For each data |
| source listed in the config, if a corresponding name (`"linux.ftrace"` in the |
| example below) was registered, the service will ask the producer process to |
| start that data source, passing it the raw bytes of the |
| [`DataSourceConfig` subsection][dss] verbatim to the data source (See |
| backward/forward compat section below). |
| |
| ![TraceConfig diagram](/docs/images/trace_config.png) |
| |
| [dss]: /docs/reference/trace-config-proto.autogen#DataSourceConfig |
| |
| ## Buffers |
| |
| The buffer sections define the number, size and policy of the in-memory buffers |
| owned by the tracing service. It looks as follows: |
| |
| ```protobuf |
| # Buffer #0 |
| buffers { |
| size_kb: 4096 |
| fill_policy: RING_BUFFER |
| } |
| |
| # Buffer #1 |
| buffers { |
| size_kb: 8192 |
| fill_policy: DISCARD |
| } |
| ``` |
| |
| Each buffer has a fill policy which is either: |
| |
| * RING_BUFFER (default): the buffer behaves like a ring buffer and writes when |
| full will wrap over and replace the oldest trace data in the buffer. |
| |
| * DISCARD: the buffer stops accepting data once full. Further write attempts are |
| dropped. |
| |
| WARNING: DISCARD can have unexpected side-effect with data sources that commit |
| data at the end of the trace. |
| |
| A trace config must define at least one buffer to be valid. In the simplest case |
| all data sources will write their trace data into the same buffer. |
| |
| While this is |
| fine for most basic cases, it can be problematic in cases where different data |
| sources write at significantly different rates. |
| |
| For instance, imagine a trace config that enables both: |
| |
| 1. The kernel scheduler tracer. On a typical Android phone this records |
| ~10000 events/second, writing ~1 MB/s of trace data into the buffer. |
| |
| 2. Memory stat polling. This data source writes the contents of /proc/meminfo |
| into the trace buffer and is configured to poll every 5 seconds, writing |
| ~100 KB per poll interval. |
| |
| If both data sources are configured to write into the same buffer and such |
| buffer is set to 4MB, most traces will contain only one memory snapshot. There |
| are very good chances that most traces won't contain any memory snapshot at all, |
| even if the 2nd data sources was working perfectly. |
| This is because during the 5 s. polling interval, the scheduler data source can |
| end up filling the whole buffer, pushing the memory snapshot data out of the |
| buffer. |
| |
| ## Dynamic buffer mapping |
| |
| Data-source <> buffer mappings are dynamic in Perfetto. |
| In the simplest case a tracing session can define only one buffer. By default, |
| all data sources will record data into that one buffer. |
| |
| In cases like the example above, it might be preferable separating these data |
| sources into different buffers. |
| This can be achieved with the `target_buffer` field of the TraceConfig. |
| |
| ![Buffer mapping](/docs/images/trace_config_buffer_mapping.png) |
| |
| Can be achieved with: |
| |
| ```protobuf |
| data_sources { |
| config { |
| name: "linux.ftrace" |
| target_buffer: 0 # <-- This goes into buffer 0. |
| ftrace_config { ... } |
| } |
| } |
| |
| data_sources: { |
| config { |
| name: "linux.sys_stats" |
| target_buffer: 1 # <-- This goes into buffer 1. |
| sys_stats_config { ... } |
| } |
| } |
| |
| data_sources: { |
| config { |
| name: "android.heapprofd" |
| target_buffer: 1 # <-- This goes into buffer 1 as well. |
| heapprofd_config { ... } |
| } |
| } |
| ``` |
| |
| ## PBTX vs binary format |
| |
| There are two ways to pass the trace config when using the `perfetto` cmdline |
| client format: |
| |
| #### Text format |
| |
| It is the preferred format for human-driven workflows and exploration. It |
| allows to pass directly the text file in the PBTX (ProtoBuf TeXtual |
| representation) syntax, for the schema defined in the |
| [trace_config.proto](/protos/perfetto/config/trace_config.proto) |
| (see [reference docs](/docs/reference/trace-config-proto.autogen)) |
| |
| When using this mode pass the `--txt` flag to `perfetto` to indicate the config |
| should be interpreted as a PBTX file: |
| |
| ```bash |
| perfetto -c /path/to/config.pbtx --txt -o trace_file.perfetto-trace |
| ``` |
| |
| NOTE: The `--txt` option has been introduced only in Android 10 (Q). Older |
| versions support only the binary format. |
| |
| WARNING: Do not use the text format for machine-to-machine interaction |
| benchmark, scripts and tools) as it's more prone to breakages (e.g. if a field |
| is renamed or an enum is turned into an integer) |
| |
| #### Binary format |
| |
| It is the preferred format for machine-to-machine (M2M) interaction. It involves |
| passing the protobuf-encoded binary of the TraceConfig message. |
| This can be obtained passing the PBTX in input to the protobuf's `protoc` |
| compiler (which can be downloaded |
| [here](https://github.com/protocolbuffers/protobuf/releases)). |
| |
| ```bash |
| cd ~/code/perfetto # external/perfetto in the Android tree. |
| |
| protoc --encode=perfetto.protos.TraceConfig \ |
| -I. protos/perfetto/config/perfetto_config.proto \ |
| < config.txpb \ |
| > config.bin |
| ``` |
| |
| and then passing it to perfetto as follows, without the `--txt` argument: |
| |
| ```bash |
| perfetto -c config.bin -o trace_file.perfetto-trace |
| ``` |
| |
| ## {#long-traces} Streaming long traces |
| |
| By default Perfetto keeps the full trace buffer(s) in memory and writes it into |
| the destination file (the `-o` cmdline argument) only at the end of the tracing |
| session. This is to reduce the perf-intrusiveness of the tracing system. |
| This, however, limits the max size of the trace to the physical memory size of |
| the device, which is often too limiting. |
| |
| In some cases (e.g., benchmarks, hard to repro cases) it is desirable to capture |
| traces that are way larger than that, at the cost of extra I/O overhead. |
| |
| To achieve that, Perfetto allows to periodically write the trace buffers into |
| the target file (or stdout) using the following TraceConfig fields: |
| |
| * `write_into_file (bool)`: |
| When true periodically drains the trace buffers into the output |
| file. When this option is enabled, the userspace buffers need to be just |
| big enough to hold tracing data between two write periods. |
| The buffer sizing depends on the activity of the device. |
| The data rate of a typical trace is ~1-4 MB/s. So a 16MB in-memory buffer can |
| hold for up write periods of ~4 seconds before starting to lose data. |
| |
| * `file_write_period_ms (uint32)`: |
| Overrides the default drain period (5s). Shorter periods require a smaller |
| userspace buffer but increase the performance intrusiveness of tracing. If |
| the period given is less than 100ms, the tracing service will use a period |
| of 100ms. |
| |
| * `max_file_size_bytes (uint64)`: |
| If set, stops the tracing session after N bytes have been written. Used to |
| cap the size of the trace. |
| |
| For a complete example of a working trace config in long-tracing mode see |
| [`/test/configs/long_trace.cfg`](/test/configs/long_trace.cfg). |
| |
| Summary: to capture a long trace just set `write_into_file:true`, set a long |
| `duration_ms` and use an in-memory buffer size of 32MB or more. |
| |
| ## Data-source specific config |
| |
| Alongside the trace-wide configuration parameters, the trace config also defines |
| data-source-specific behaviors. At the proto schema level, this is defined in |
| the `DataSourceConfig` section of `TraceConfig`: |
| |
| From [data_source_config.proto](/protos/perfetto/config/data_source_config.proto): |
| |
| ```protobuf |
| message TraceConfig { |
| ... |
| repeated DataSource data_sources = 2; // See below. |
| } |
| |
| message DataSource { |
| optional protos.DataSourceConfig config = 1; // See below. |
| ... |
| } |
| |
| message DataSourceConfig { |
| optional string name = 1; |
| ... |
| optional FtraceConfig ftrace_config = 100 [lazy = true]; |
| ... |
| optional AndroidPowerConfig android_power_config = 106 [lazy = true]; |
| } |
| ``` |
| |
| Fields like `ftrace_config`, `android_power_config` are examples of data-source |
| specific configs. The tracing service will completely ignore the contents of |
| those fields and route the whole DataSourceConfig object to any data source |
| registered with the same name. |
| |
| The `[lazy=true]` marker has a special implication in the |
| [protozero](/docs/design-docs/protozero.md) code generator. Unlike standard |
| nested messages, it generates raw accessors (e.g., |
| `const std::string& ftrace_config_raw()` instead of |
| `const protos::FtraceConfig& ftrace_config()`). This is to avoid injecting too |
| many `#include` dependencies and avoiding binary size bloat in the code that |
| implements data sources. |
| |
| #### A note on backwards/forward compatibility |
| The tracing service will route the raw binary blob of the `DataSourceConfig` |
| message to the data sources with a matching name, without attempting to decode |
| and re-encode it. If the `DataSourceConfig` section of the trace config contains |
| a new field that didn't exist at the time when the service was built, the |
| service will still pass the `DataSourceConfig` through to the data source. |
| This allows to introduced new data sources without needing the service to |
| know anything about them upfront. |
| |
| TODO: we are aware of the fact that today extending the `DataSourceConfig` with |
| a custom proto requires changing the `data_source_config.proto` in the Perfetto |
| repo, which is unideal for external projects. The long-term plan is to reserve |
| a range of fields for non-upstream extensions and provide generic templated |
| accessors for client code. Until then, we accept patches upstream to introduce |
| ad-hoc configurations for your own data sources. |
| |
| ## Multi-process data sources |
| |
| Some data sources are singletons. E.g., in the case of scheduler tracing that |
| Perfetto ships on Android, there is only data source for the whole system, |
| owned by the `traced_probes` service. |
| |
| However, in the general case multiple processes can advertise the same data |
| source. This is the case, for instance, when using the |
| [Perfetto SDK](/docs/instrumentation/tracing-sdk.md) for userspace |
| instrumentation. |
| |
| If this happens, when starting a tracing session that specifies that data |
| source in the trace config, Perfetto by default will ask all processes that |
| advertise that data source to start it. |
| |
| In some cases it might be desirable to further limit the enabling of the data |
| source to a specific process (or set of processes). That is possible through the |
| `producer_name_filter` and `producer_name_regex_filter`. |
| |
| NOTE: the typical Perfetto run-time model is: one process == one Perfetto |
| Producer; one Producer typically hosts multiple data sources. |
| |
| When those filters are set, the Perfetto tracing service will activate the data |
| source only in the subset of producers matching the filter. |
| |
| Example: |
| |
| ```protobuf |
| buffers { |
| size_kb: 4096 |
| } |
| |
| data_sources { |
| config { |
| name: "track_event" |
| } |
| # Enable the data source only on Chrome and Chrome canary. |
| producer_name_filter: "com.android.chrome" |
| producer_name_filter: "com.google.chrome.canary" |
| } |
| ``` |
| |
| ## Triggers |
| |
| In nominal conditions, a tracing session has a lifecycle that simply matches the |
| invocation of the `perfetto` cmdline client: trace data recording starts when |
| the TraceConfig is passed to `perfetto` and ends when either the |
| `TraceConfig.duration_ms` has elapsed, or when the cmdline client terminates. |
| |
| Perfetto supports an alternative mode of either starting or stopping the trace |
| which is based on triggers. The overall idea is to declare in the trace config |
| itself: |
| |
| * A set of triggers, which are just free-form strings. |
| * Whether a given trigger should cause the trace to be started or stopped, and |
| the start/stop delay. |
| |
| Why using triggers? Why can't one just start perfetto or kill(SIGTERM) it when |
| needed? The rationale of all this is the security model: in most Perfetto |
| deployments (e.g., on Android) only privileged entities (e.g., adb shell) can |
| configure/start/stop tracing. Apps are unprivileged in this sense and they |
| cannot control tracing. |
| |
| Triggers offer a way to unprivileged apps to control, in a limited fashion, the |
| lifecycle of a tracing session. The conceptual model is: |
| |
| * The privileged Consumer (see |
| [_Service model_](/docs/concepts/service-model.md)), i.e. the entity |
| that is normally authorized to start tracing (e.g., adb shell in Android), |
| declares upfront what are the possible trigger names for the trace and what |
| they will do. |
| * Unprivileged entities (any random app process) can activate those triggers. |
| Unprivileged entities don't get a say on what the triggers will do, they only |
| communicate that an event happened. |
| |
| Triggers can be signaled via the cmdline util |
| |
| ```bash |
| /system/bin/trigger_perfetto "trigger_name" |
| ``` |
| |
| (or also by starting an independent trace session which uses only the |
| `activate_triggers: "trigger_name"` field in the config) |
| |
| There are two types of triggers: |
| |
| #### Start triggers |
| |
| Start triggers allow activating a tracing session only after some significant |
| event has happened. Passing a trace config that has `START_TRACING` trigger |
| causes the tracing session to stay idle (i.e. not recording any data) until either |
| the trigger is hit or the `trigger_timeout_ms` timeout is hit. |
| |
| `trace_duration_ms` and triggered traces can not be used at the same time. |
| |
| Example config: |
| ```protobuf |
| # If the "myapp_is_slow" is hit, the trace starts recording data and will be |
| # stopped after 5s. |
| trigger_config { |
| trigger_mode: START_TRACING |
| triggers { |
| name: "myapp_is_slow" |
| stop_delay_ms: 5000 |
| } |
| # If no trigger is hit, the trace will end without having recorded any data |
| # after 30s. |
| trigger_timeout_ms: 30000 |
| } |
| |
| # The rest of the config is as usual. |
| buffers { ... } |
| data_sources { ... } |
| ``` |
| |
| #### Stop triggers |
| |
| STOP_TRACING triggers allow to prematurely finalize a trace when the trigger is |
| hit. In this mode the trace starts immediately when the `perfetto` client is |
| invoked (like in nominal cases). The trigger acts as a premature finalization |
| signal. |
| |
| This can be used to use perfetto in flight-recorder mode. By starting a trace |
| with buffers configured in `RING_BUFFER` mode and `STOP_TRACING` triggers, |
| the trace will be recorded in a loop and finalized when the culprit event is |
| detected. This is key for events where the root cause is in the recent past |
| (e.g., the app detects a slow scroll or a missing frame). |
| |
| Example config: |
| ```protobuf |
| # If no trigger is hit, the trace will end after 30s. |
| trigger_timeout_ms: 30000 |
| |
| # If the "missed_frame" is hit, the trace is stopped after 1s. |
| trigger_config { |
| trigger_mode: STOP_TRACING |
| triggers { |
| name: "missed_frame" |
| stop_delay_ms: 1000 |
| } |
| } |
| |
| # The rest of the config is as usual. |
| buffers { ... } |
| data_sources { ... } |
| ``` |
| |
| ## Android |
| |
| On Android, there are some caveats around using `adb shell` |
| |
| * Ctrl+C, which normally causes a graceful termination of the trace, is not |
| propagated by ADB when using `adb shell perfetto` but only when using an |
| interactive PTY-based session via `adb shell`. |
| * On non-rooted devices before Android 12, the config can only be passed as |
| `cat config | adb shell perfetto -c -` (-: stdin) because of over-restrictive |
| SELinux rules. Since Android 12 `/data/misc/perfetto-configs` can be used for |
| storing configs. |
| * On devices before Android 10, adb cannot directly pull |
| `/data/misc/perfetto-traces`. Use |
| `adb shell cat /data/misc/perfetto-traces/trace > trace` to work around. |
| * When capturing longer traces, e.g. in the context of benchmarks or CI, use |
| `PID=$(perfetto --background)` and then `kill $PID` to stop. |
| |
| |
| ## Other resources |
| |
| * [TraceConfig Reference](/docs/reference/trace-config-proto.autogen) |
| * [Buffers and dataflow](/docs/concepts/buffers.md) |