# Quickstart: SQL-based analysis and trace-based metrics

_This quickstart explains how to use `trace_processor` as well as its Python API to 
programmatically query the trace contents through SQL and compute trace-based metrics._

## Trace Processor

TraceProcessor is a multi-format trace importing and query engine based on
SQLite. It comes both as a C++ library and as a standalone executable:
`trace_processor_shell` (or just `trace_processor`).

### Setup

```bash
# Download prebuilts (Linux and Mac only)
curl -LO https://get.perfetto.dev/trace_processor
chmod +x ./trace_processor

# Start the interactive shell
./trace_processor trace.pftrace

# Start a local trace processor instance to replace wasm module in the UI
./trace_processor trace.pftrace --http
```

NOTE: In HTTP mode the trace will be loaded into the `trace_processor` and
      the UI will connect and issue queries over TCP. This can allow
      arbitrary sized traces to be loaded since there are no memory
      constraints, unlike the WASM module. In addition, this can improve
      performance in the UI as it issues SQL queries.

See [Trace Processor docs](/docs/analysis/trace-processor.md) for the full
TraceProcessor guide.

### Sample queries

For more exhaustive examples see the _SQL_ section of the various _Data sources_
docs.

#### Slices

Slices are stackable events which have name and span some duration of time.

![](/docs/images/slices.png "Example of slices in the UI")

```
> SELECT ts, dur, name FROM slice
ts                   dur                  name
-------------------- -------------------- ---------------------------
     261187017446933               358594 eglSwapBuffersWithDamageKHR
     261187017518340                  357 onMessageReceived
     261187020825163                 9948 queueBuffer
     261187021345235                  642 bufferLoad
     261187121345235                  153 query
     ...
```

#### Counters

Counters are events with a value which changes over time.

![](/docs/images/counters.png "Example of counters in the UI")

```
> SELECT ts, value FROM counter
ts                   value
-------------------- --------------------
     261187012149954          1454.000000
     261187012399172          4232.000000
     261187012447402         14304.000000
     261187012535839         15490.000000
     261187012590890         17490.000000
     261187012590890         16590.000000
...
```

#### Scheduler slices

Scheduler slices indicate which thread was scheduled on which CPU at which time.

![](/docs/images/sched-slices.png "Example of scheduler slices in the UI")

```
> SELECT ts, dur, cpu, utid FROM sched
ts                   dur                  cpu                  utid
-------------------- -------------------- -------------------- --------------------
     261187012170489               267188                    0                  390
     261187012170995               247153                    1                  767
     261187012418183                12812                    2                 2790
     261187012421099               220000                    6                  683
     261187012430995                72396                    7                 2791
...
```

### Trace-based metrics

Trace Processor offers also a higher-level query interface that allows to run
pre-baked queries, herein called "metrics". Metrics are generally curated by
domain experts, often the same people who add the instrumentation points in the
first lace, and output structured JSON/Protobuf/text.
Metrics allow to get a summarized view of the trace without having to type any
SQL or having to load the trace in the UI.

The metrics` schema files live in the
[/protos/perfetto/metrics](/protos/perfetto/metrics/) directory.
The corresponding SQL queries live in
[/src/trace_processor/metrics](/src/trace_processor/metrics/).

#### Run a single metric

Let's run the [`android_cpu`](/protos/perfetto/metrics/android/cpu_metric.proto)
metric. This metrics computes the total CPU time and the total cycles
(CPU frequency * time spent running at that frequency) for each process in the
trace, breaking it down by CPU (_core_) number.

```protobuf
./trace_processor --run-metrics android_cpu trace.pftrace

android_cpu {
  process_info {
    name: "/system/bin/init"
    threads {
      name: "init"
      core {
        id: 1
        metrics {
          mcycles: 1
          runtime_ns: 570365
          min_freq_khz: 1900800
          max_freq_khz: 1900800
          avg_freq_khz: 1902017
        }
      }
      core {
        id: 3
        metrics {
          mcycles: 0
          runtime_ns: 366406
          min_freq_khz: 1900800
          max_freq_khz: 1900800
          avg_freq_khz: 1902908
        }
      }
      ...
    }
    ...
  }
  process_info {
    name: "/system/bin/logd"
    threads {
      name: "logd.writer"
      core {
        id: 0
        metrics {
          mcycles: 8
          runtime_ns: 33842357
          min_freq_khz: 595200
          max_freq_khz: 1900800
          avg_freq_khz: 1891825
        }
      }
      core {
        id: 1
        metrics {
          mcycles: 9
          runtime_ns: 36019300
          min_freq_khz: 1171200
          max_freq_khz: 1900800
          avg_freq_khz: 1887969
        }
      }
      ...
    }
    ...
  }
  ...
}
```

#### Running multiple metrics

Multiple metrics can be flagged using comma separators to the `--run-metrics`
flag. This will output a text proto with the combined result of running both
metrics.

```protobuf
$ ./trace_processor --run-metrics android_mem,android_cpu trace.pftrace

android_mem {
  process_metrics {
    process_name: ".dataservices"
    total_counters {
      anon_rss {
        min: 19451904
        max: 19890176
        avg: 19837548.157829277
      }
      file_rss {
        min: 25804800
        max: 25829376
        avg: 25827909.957489081
      }
      swap {
        min: 9289728
        max: 9728000
        avg: 9342355.8421707246
      }
      anon_and_swap {
        min: 29179904
        max: 29179904
        avg: 29179904
      }
    }
    ...
  }
  ...
}
android_cpu {
  process_info {
    name: "/system/bin/init"
    threads {
      name: "init"
      core {
        id: 1
        metrics {
          mcycles: 1
          runtime_ns: 570365
          min_freq_khz: 1900800
          max_freq_khz: 1900800
          avg_freq_khz: 1902017
        }
      }
      ...
    }
    ...
  }
  ...
}
```

#### JSON and binary output

The trace processor also supports binary protobuf and JSON as alternative output
formats. This is useful when the intended reader is an offline tool.

Both single and multiple metrics are supported as with proto text output.

```
./trace_processor --run-metrics android_mem --metrics-output=binary trace.pftrace
<binary protobuf output>

./trace_processor --run-metrics android_mem,android_cpu --metrics-output=json trace.pftrace
{
  "android_mem": {
    "process_metrics": [
      {
        "process_name": ".dataservices",
        "total_counters": {
          "anon_rss": {
            "min": 19451904.000000,
            "max": 19890176.000000,
            "avg": 19837548.157829
          },
          "file_rss": {
            "min": 25804800.000000,
            "max": 25829376.000000,
            "avg": 25827909.957489
          },
          "swap": {
            "min": 9289728.000000,
            "max": 9728000.000000,
            "avg": 9342355.842171
          },
          "anon_and_swap": {
            "min": 29179904.000000,
            "max": 29179904.000000,
            "avg": 29179904.000000
          }
        },
        ...
      },
      ...
    ]
  }
  "android_cpu": {
    "process_info": [
      {
        "name": "\/system\/bin\/init",
        "threads": [
          {
            "name": "init",
            "core": [
              {
                "id": 1,
                "metrics": {
                  "mcycles": 1,
                  "runtime_ns": 570365,
                  "min_freq_khz": 1900800,
                  "max_freq_khz": 1900800,
                  "avg_freq_khz": 1902017
                }
              },
              ...
            ]
            ...
          }
          ...
        ]
        ...
      },
      ...
    ]
    ...
  }
}
```

## Python API

The API can be run without requiring the `trace_processor` binary to be
downloaded or installed.

### Example functions
See the Python API section of
[Trace Processor (SQL)](/docs/analysis/trace-processor.md) to get
more details on all available functions.

#### Query
```python
from trace_processor.api import TraceProcessor
tp = TraceProcessor(file_path='trace.pftrace')

qr_it = tp.query('SELECT name FROM slice')
for row in qr_it:
  print(row.name)
```
**Output**
```
eglSwapBuffersWithDamageKHR
onMessageReceived
queueBuffer
bufferLoad
query
...
```
#### Query as Pandas DataFrame
```python
from trace_processor.api import TraceProcessor
tp = TraceProcessor(file_path='trace.pftrace')

qr_it = tp.query('SELECT ts, name FROM slice')
qr_df = qr_it.as_pandas()
print(qr_df.to_string())
```
**Output**
```
ts                   name
-------------------- ---------------------------
     261187017446933 eglSwapBuffersWithDamageKHR
     261187017518340 onMessageReceived
     261187020825163 queueBuffer
     261187021345235 bufferLoad
     261187121345235 query
     ...
```
#### Metric
```python
from trace_processor.api import TraceProcessor
tp = TraceProcessor(file_path='trace.pftrace')

cpu_metrics = tp.metric(['android_cpu'])
print(cpu_metrics)
```
**Output**
```
metrics {
  android_cpu {
    process_info {
      name: "/system/bin/init"
      threads {
        name: "init"
        core {
          id: 1
          metrics {
            mcycles: 1
            runtime_ns: 570365
            min_freq_khz: 1900800
            max_freq_khz: 1900800
            avg_freq_khz: 1902017
          }
        }
        core {
          id: 3
          metrics {
            mcycles: 0
            runtime_ns: 366406
            min_freq_khz: 1900800
            max_freq_khz: 1900800
            avg_freq_khz: 1902908
          }
        }
        ...
      }
      ...
    }
    ...
  }
}
```

## Next steps

There are several options for exploring more of the trace analysis features
Perfetto provides:

* The [trace conversion quickstart](/docs/quickstart/traceconv.md) gives an
  overview on how to convert Perfetto traces to legacy formats to integrate with
  existing tooling.
* The [Trace Processor documentation](/docs/analysis/trace-processor.md) gives
  more information about how to work with trace processor including details on
  how to write queries and how tables in trace processor are organized.
* The [metrics documentation](/docs/analysis/metrics.md) gives a more in-depth
  look into metrics including a short walkthrough on how to build an
  experimental metric from scratch.
* The [SQL table reference](/docs/analysis/sql-tables.autogen) gives a
  comprehensive guide to the all the available tables in trace processor.
* The [common tasks](/docs/contributing/common-tasks.md) page gives a list of
  steps on how new metrics can be added to the trace processor.