docs/learning-more/symbolization.md

# Symbolization and deobfuscation

This document describes how to turn raw instruction addresses and obfuscated
Java/Kotlin names in a collected trace into human-readable function names,
source locations, and class/method names.

The right approach depends on **what kind of trace you have**, so this page is
organised around that question. Two definitions used throughout:

- **Symbolization**: mapping native instruction addresses back to function
  names, source files, and line numbers, using the unstripped ELF binaries (or
  equivalent Breakpad symbol files) that were loaded in the profiled process.
- **Deobfuscation**: mapping the obfuscated Java/Kotlin names emitted by
  R8/ProGuard (e.g. `fsd.a`) back to the original identifiers, using the
  `mapping.txt` produced at build time.

## Which workflow do you need? {#which-workflow}

Match your trace to one of the categories below and follow the link. Picking the
wrong workflow is the most common reason symbols "don't work". The key rule of
thumb: **userspace** symbols are resolved offline on the host (`traceconv
bundle`), while **kernel** symbols are always resolved at record time on the
device (Perfetto never stores absolute kernel addresses, to avoid disclosing
[KASLR](https://en.wikipedia.org/wiki/Address_space_layout_randomization)).

| Your trace contains… | Examples | What you need |
| --- | --- | --- |
| **Callstacks** | Native heap profiler, `traced_perf` / Linux perf CPU sampling, ART heap dumps | [Symbolization & deobfuscation](#callstacks). Userspace frames are resolved offline (`traceconv bundle`); kernel frames are symbolized on-device automatically. |
| **Kernel ftrace events** | `function_graph` tracing, `sched_blocked_reason`, kprobes | [Record-time `symbolize_ksyms`](#ftrace). These addresses **cannot** be symbolized after the fact. |
| **Userspace event names** | atrace slice names, ART method tracing | [Not currently supported](#userspace-event-names) for offline deobfuscation; emit readable names at instrumentation time. |

## Callstacks: symbolization and deobfuscation {#callstacks}

This applies to any data source that captures callstacks: the native heap
profiler, the perf-based CPU profiler (`traced_perf` and imported Linux `perf`
data), and the ART allocation profiler.

These data sources record raw **userspace** instruction addresses (and, on
Android, obfuscated Java/Kotlin frames), which you resolve on the host **after
recording** with the steps below. You do **not** need to re-record to get
userspace symbols or deobfuscated names, as long as you still have the matching
binaries and mapping files.

Callstacks can also contain **kernel** frames, which are handled differently; see
[Kernel frames in callstacks](#callstack-kernel-frames) at the end of this
section.

### {#option-1-traceconv-bundle} Option 1: `traceconv bundle` (recommended)

`traceconv bundle` is a one-shot command that takes a trace and produces an
**enriched trace**: the original trace plus all the symbol and deobfuscation
data needed to analyse it, packaged together in a single file.

```bash
traceconv bundle input.perfetto-trace enriched-trace
```

The enriched trace can be opened in the [Perfetto UI](https://ui.perfetto.dev)
or in `trace_processor_shell` like any other trace, with symbols and
deobfuscated names already applied.

NOTE: As an implementation detail, the enriched trace is currently packaged as a
TAR archive containing the original trace, native symbol packets, and
Java/Kotlin deobfuscation packets. The UI and `trace_processor_shell` read this
format transparently, so you normally don't need to unpack it yourself.

**Requirements:**

- `llvm-symbolizer` on `$PATH` for native symbolization to produce function
  names and line numbers (`sudo apt install llvm` on Debian/Ubuntu).
- Input and output must be file paths; stdin/stdout are not supported.
- Matching unstripped binaries / Breakpad symbols on disk (Build IDs must match
  what was recorded on device).
- For Java/Kotlin: the `mapping.txt` produced by the build that ran on the
  device.

#### Automatic path discovery

The main advantage over
[Option 2](#option-2-legacy-traceconv-symbolize-deobfuscate) is that `bundle`
looks for symbols and mapping files in all the obvious places without
configuration. It searches:

- The AOSP build output (`$ANDROID_PRODUCT_OUT/symbols`) when running inside a
  `lunch`-ed AOSP checkout.
- Standard system debug directories (`$HOME/.debug`, `/usr/lib/debug`).
- Absolute library paths recorded in the trace's `stack_profile_mapping` (useful
  when profiling on the same machine you are analysing on).
- The standard Android Gradle project layout for ProGuard/R8 mapping files
  (`./app/build/outputs/mapping/<variant>/mapping.txt`).

#### Supplementing discovery with flags

When auto-discovery isn't enough:

```bash
traceconv bundle \
  --symbol-paths /path/to/symbols1,/path/to/symbols2 \
  --proguard-map com.example.app=/path/to/mapping.txt \
  --verbose \
  input.perfetto-trace enriched-trace
```

The properties of the `bundle` flags are:

- `--symbol-paths PATH1,PATH2,...`: additional directories to search for native
  symbols (in addition to the auto-discovered ones).
- `--no-auto-symbol-paths`: disable auto-discovery of native symbol paths. Only
  paths given via `--symbol-paths` are searched.
- `--proguard-map [pkg=]PATH`: additional ProGuard/R8 `mapping.txt` to apply for
  Java/Kotlin deobfuscation. Repeat the flag for multiple maps. The optional
  `pkg=` prefix scopes a map to a specific Java package.
- `--no-auto-proguard-maps`: disable auto-discovery of ProGuard/R8 mapping files
  (e.g. the standard Android Gradle layout). Only maps given via
  `--proguard-map` are applied.
- `--verbose`: print every path tried and every library looked up &mdash; useful
  when debugging "could not find" errors.

### {#option-2-legacy-traceconv-symbolize-deobfuscate} Option 2: Legacy `traceconv symbolize` / `deobfuscate`

NOTE: This flow is kept for backwards compatibility with existing scripts and
CI pipelines that already depend on it. For new usage, always prefer
[Option 1](#option-1-traceconv-bundle) &mdash; it is simpler, has
auto-discovery, and works on non-Perfetto trace formats.

The older `traceconv symbolize` and `traceconv deobfuscate` subcommands
produce standalone symbol and deobfuscation files driven entirely by
environment variables, which must then be concatenated onto the trace by
hand.

#### Native symbolization

All tools (`traceconv`, `trace_processor_shell`, the `heap_profile` script)
honour the `PERFETTO_BINARY_PATH` environment variable:

```bash
PERFETTO_BINARY_PATH=somedir tools/heap_profile android --name ${NAME}
```

To produce a standalone symbol file for a trace you already collected:

```bash
PERFETTO_BINARY_PATH=somedir traceconv symbolize raw-trace > symbols
```

Alternatively, set `PERFETTO_SYMBOLIZER_MODE=index` and the symbolizer will
recursively index the directory for ELF files by Build ID, so filenames do not
need to match.

#### Java/Kotlin deobfuscation

Provide ProGuard/R8 maps via `PERFETTO_PROGUARD_MAP`, using the format
`packagename=map_filename[:packagename=map_filename...]`:

```bash
PERFETTO_PROGUARD_MAP=com.example.pkg1=foo.txt:com.example.pkg2=bar.txt \
  ./tools/heap_profile android -n com.example.app
```

To produce a standalone deobfuscation file for an existing trace:

```bash
PERFETTO_PROGUARD_MAP=com.example.pkg=proguard_map.txt \
  traceconv deobfuscate ${TRACE} > deobfuscation_map
```

#### Attaching the output to a trace

Both `symbols` and `deobfuscation_map` above are serialized `TracePacket`
protos, so for a **Perfetto protobuf trace** you can simply concatenate them:

```bash
cat ${TRACE} symbols > symbolized-trace
cat ${TRACE} deobfuscation_map > deobfuscated-trace
# or both:
cat ${TRACE} symbols deobfuscation_map > enriched-trace
```

The `tools/heap_profile` script does this automatically in its output directory
when `PERFETTO_BINARY_PATH` is set.

**Limitations:**

- The concatenation trick **only works for Perfetto protobuf traces**. Other
  trace formats (Chrome JSON, systrace, Firefox profile, etc.) cannot have
  `TracePacket` bytes appended this way. For those formats, use
  [Option 1](#option-1-traceconv-bundle) and load the symbols via
  `trace_processor_shell`.
- You must manage `PERFETTO_BINARY_PATH` / `PERFETTO_PROGUARD_MAP` by hand; none
  of the auto-discovery from Option 1 applies.

### Symbol lookup order

For each native mapping in the trace, the symbolizer looks for a file with
matching Build ID. For each search path `P`, it tries (in order):

1. Absolute path of the library file relative to `P`.
2. Same, with `base.apk!` stripped from the filename.
3. Basename of the library file relative to `P`.
4. Basename, with `base.apk!` stripped.
5. `P/.build-id/<first 2 hex digits>/<rest>.debug` (the standard
   [Fedora Build ID layout](https://fedoraproject.org/wiki/RolandMcGrath/BuildID#Find_files_by_build_ID)).

For example, `/system/lib/base.apk!foo.so` with build id `abcd1234...` is looked
up under a symbol path `P` at:

1. `P/system/lib/base.apk!foo.so`
2. `P/system/lib/foo.so`
3. `P/base.apk!foo.so`
4. `P/foo.so`
5. `P/.build-id/ab/cd1234...debug`

The first file with a matching Build ID wins. If the Build ID on disk differs
from the one recorded in the trace, the file is skipped.

### Using symbolization/deobfuscation from a C++ library

There is currently **no stable public C++ API** for performing symbolization or
deobfuscation in-process. The underlying implementation exists (`TraceToBundle`
in `src/traceconv/trace_to_bundle.h`, backed by `EnrichTrace` in
`src/trace_processor/util/trace_enrichment/trace_enrichment.h`), but it lives
under `src/` rather than `include/` and is not part of the public API surface.

If you need this, please +1 on
[GitHub issue #5534](https://github.com/google/perfetto/issues/5534) so we can
gauge demand and prioritise.

### Troubleshooting

#### Could not find library

When symbolizing a profile you may see messages like:

```text
Could not find /data/app/invalid.app-wFgo3GRaod02wSvPZQ==/lib/arm64/somelib.so
(Build ID: 44b7138abd5957b8d0a56ce86216d478).
```

Check that `somelib.so` exists somewhere under one of the search paths
(`--symbol-paths`, `PERFETTO_BINARY_PATH`, or an auto-discovered location). Then
compare the Build ID on disk to the one reported in the message using
`readelf -n /path/to/somelib.so`. If they do not match, the copy on disk is a
different build than the one on device and cannot be used.

Re-running `traceconv bundle` with `--verbose` prints every path tried, which
usually makes it clear whether the file was missing entirely or found with the
wrong Build ID.

### Kernel frames in callstacks {#callstack-kernel-frames}

A sampled callstack can include **kernel** frames (e.g. perf sampling with
`callstack_sampling { kernel_frames: true }`). Unlike the userspace frames above,
these are symbolized **automatically on the device at record time** from
`/proc/kallsyms` &mdash; the offline tools in this section do not touch them.

For kernel frames to be named, the recording must be able to read
`/proc/kallsyms`, which requires running as root or lowering `kptr_restrict`:

```bash
echo 0 | sudo tee /proc/sys/kernel/kptr_restrict
```

If kernel frames show as hex addresses, this is a record-time permissions issue
and you have to re-record. This is the same KASLR constraint as for
[kernel ftrace events](#ftrace) below, but note the two use different
mechanisms: callstack kernel frames do **not** use the `symbolize_ksyms` ftrace
option &mdash; that flag only affects ftrace events.

## Kernel ftrace events: `symbolize_ksyms` {#ftrace}

If you are doing **system tracing** and seeing raw hexadecimal addresses where
you expected kernel function names &mdash; for example in
[function graph tracing](/docs/data-sources/funcgraph.md), in the
`blocked_function` field of an uninterruptible-sleep
[scheduling blockage](/docs/case-studies/scheduling-blockages.md), or in kprobe
events &mdash; the fix is **not** offline symbolization.

These kernel addresses are resolved **at record time** by enabling
`symbolize_ksyms` in the ftrace config:

```protobuf
data_sources: {
    config {
        name: "linux.ftrace"
        ftrace_config {
            symbolize_ksyms: true
            # ... your ftrace_events / function_graph config ...
        }
    }
}
```

This reads `/proc/kallsyms` on the device and embeds the (mangled) symbol map in
the trace. It requires that either `traced_probes` runs as root or
`kptr_restrict` has been lowered manually.

WARNING: `traceconv bundle` and the offline symbolizers above **cannot** recover
kernel symbols. Perfetto deliberately does not store absolute kernel addresses
in the trace, because doing so would defeat
[KASLR](https://en.wikipedia.org/wiki/Address_space_layout_randomization) and
disclose the kernel memory layout. The symbol names are mangled on device so
this works without leaking absolute addresses. If you forgot to set
`symbolize_ksyms`, you have to re-record.

This flag applies only to ftrace **events**. Kernel frames captured inside
sampled callstacks are handled separately; see
[Kernel frames in callstacks](#callstack-kernel-frames).

## Userspace event names: atrace and ART method tracing {#userspace-event-names}

Some data sources record human-readable **name strings** rather than addresses
or stack frames. When those strings are obfuscated (e.g. an R8-obfuscated class
name), there is **no offline mechanism to deobfuscate them** &mdash; the name
must be emitted in a readable form at instrumentation time. This is distinct from
the Java/Kotlin **stack-frame** deobfuscation in
[the callstacks section](#callstacks), which applies only to heap dumps and
sampled callstacks.

This affects two cases today:

- **atrace / userspace slice names**: [atrace](/docs/data-sources/atrace.md)
  slice names (and other strings that ended up in a `TRACE_EVENT` literal) are
  recorded verbatim. There is no post-hoc mapping step.
- **ART method tracing**: the method names captured by ART method tracing are not
  run through the ProGuard/R8 deobfuscation path, so obfuscated builds will show
  obfuscated method names.

A `mapping.txt`-based deobfuscation path for these is in principle possible but
not currently implemented. Support is under discussion; see
[GitHub issue #6391](https://github.com/google/perfetto/issues/6391) for context
and to register interest.