docs/learning-more/symbolization.md

# Symbolization and deobfuscation

This document describes how to turn raw instruction addresses and obfuscated
Java/Kotlin names in a collected trace into human-readable function names,
source locations, and class/method names.

The right approach depends on **what kind of trace you have**, so this page is
organised around that question. Two definitions used throughout:

- **Symbolization**: mapping native instruction addresses back to function
  names, source files, and line numbers, using the unstripped ELF binaries (or
  equivalent Breakpad symbol files) that were loaded in the profiled process.
- **Deobfuscation**: mapping the obfuscated Java/Kotlin names emitted by
  R8/ProGuard (e.g. `fsd.a`) back to the original identifiers, using the
  `mapping.txt` produced at build time.

## Fetch debug files using debuginfod

If you have a trace with native build IDs and access to a debuginfod server,
you can create a symbolized bundle without supplying local binaries. Install
`curl` and `llvm-symbolizer`, then run:

```sh
trace_processor bundle --debuginfod \
  --debuginfod-urls "https://your-debuginfod-server.example" \
  input.pftrace output.tar
```

If `DEBUGINFOD_URLS` is already configured, only `--debuginfod` is needed.
Open the resulting bundle in the UI. Check the reported unresolved-frame count;
use `--verbose` to investigate unsuccessful lookups. Downloads are cached for
later runs. Use `--debuginfod-cache-path PATH` to choose a different cache.

See the [CLI reference](/docs/reference/trace-processor-cli.md#debuginfod) for
precedence, timeouts, cache layout, and output controls.

## Which workflow do you need? {#which-workflow}

Match your trace to one of the categories below and follow the link. Picking the
wrong workflow is the most common reason symbols "don't work". The key rule of
thumb: **userspace** symbols are resolved offline on the host (`trace_processor
bundle`), while **kernel** symbols are always resolved at record time on the
device (Perfetto never stores absolute kernel addresses, to avoid disclosing
[KASLR](https://en.wikipedia.org/wiki/Address_space_layout_randomization)).

| Your trace contains… | Examples | What you need |
| --- | --- | --- |
| **Callstacks** | Native heap profiler, `traced_perf` / Linux perf CPU sampling, ART heap dumps | [Symbolization & deobfuscation](#callstacks). Userspace frames are resolved offline (`trace_processor bundle`); kernel frames are symbolized on-device automatically. |
| **Kernel ftrace events** | `function_graph` tracing, `sched_blocked_reason`, kprobes | [Record-time `symbolize_ksyms`](#ftrace). These addresses **cannot** be symbolized after the fact. |
| **Userspace event names** | atrace slice names, ART method tracing | [Not currently supported](#userspace-event-names) for offline deobfuscation; emit readable names at instrumentation time. |

## Callstacks: symbolization and deobfuscation {#callstacks}

This applies to any data source that captures callstacks: the native heap
profiler, the perf-based CPU profiler (`traced_perf` and imported Linux `perf`
data), and the ART allocation profiler.

These data sources record raw **userspace** instruction addresses (and, on
Android, obfuscated Java/Kotlin frames), which you resolve on the host **after
recording** with the steps below. You do **not** need to re-record to get
userspace symbols or deobfuscated names, as long as you still have the matching
binaries and mapping files.

Callstacks can also contain **kernel** frames, which are handled differently; see
[Kernel frames in callstacks](#callstack-kernel-frames) at the end of this
section.

### {#option-1-traceconv-bundle} Option 1: `trace_processor bundle` (recommended)

`trace_processor bundle` is a one-shot command that takes a trace and produces an
**enriched trace**: the original trace plus all the symbol and deobfuscation
data needed to analyse it, packaged together in a single file.

```bash
trace_processor bundle input.perfetto-trace enriched-trace
```

The enriched trace can be opened in the [Perfetto UI](https://ui.perfetto.dev)
or in `trace_processor_shell` like any other trace, with symbols and
deobfuscated names already applied.

NOTE: As an implementation detail, the enriched trace is currently packaged as a
TAR archive containing the original trace, native symbol packets, and
Java/Kotlin deobfuscation packets. The UI and `trace_processor_shell` read this
format transparently, so you normally don't need to unpack it yourself.

**Requirements:**

- `llvm-symbolizer` on `$PATH` for native symbolization to produce function
  names and line numbers (`sudo apt install llvm` on Debian/Ubuntu).
- Input and output must be file paths; stdin/stdout are not supported.
- Matching unstripped binaries / Breakpad symbols on disk (Build IDs must match
  what was recorded on device).
- For Java/Kotlin: the `mapping.txt` produced by the build that ran on the
  device.

#### Automatic path discovery

The main advantage over
[Option 2](#option-2-legacy-traceconv-symbolize-deobfuscate) is that `bundle`
looks for symbols and mapping files in all the obvious places without
configuration. It searches:

- The AOSP build output (`$ANDROID_PRODUCT_OUT/symbols`) when running inside a
  `lunch`-ed AOSP checkout.
- Standard system debug directories (`$HOME/.debug`, `/usr/lib/debug`).
- Absolute library paths recorded in the trace's `stack_profile_mapping` (useful
  when profiling on the same machine you are analysing on).
- The standard Android Gradle project layout for ProGuard/R8 mapping files
  (`./app/build/outputs/mapping/<variant>/mapping.txt`).

#### Supplementing discovery with flags

When auto-discovery isn't enough:

```bash
trace_processor bundle \
  --symbol-paths /path/to/symbols1,/path/to/symbols2 \
  --proguard-map com.example.app=/path/to/mapping.txt \
  --verbose \
  input.perfetto-trace enriched-trace
```

Point `--symbol-paths` at directories containing the matching unstripped
binaries or native debug files, for example your build's symbols directory.
Bundle searches recursively and matches build IDs, so you do not need to
recreate the device's directory layout. Pass Java/Kotlin `mapping.txt` files
separately with `--proguard-map`.

Use `--verbose` to diagnose missing symbols. To disable automatic discovery,
add `--no-auto-symbol-paths` and `--no-auto-proguard-maps`. Native paths from
`PERFETTO_BINARY_PATH` still apply; unset that variable to restrict lookup to
`--symbol-paths`.

See the [bundle command reference](/docs/reference/trace-processor-cli.md#subcommand-bundle)
for option semantics, color controls, output replacement, and exit status.

### {#option-2-legacy-traceconv-symbolize-deobfuscate} Option 2: Legacy `trace_processor util symbolize` / `util deobfuscate`

NOTE: This flow is kept for backwards compatibility with existing scripts and
CI pipelines that already depend on it. For new usage, always prefer
[Option 1](#option-1-traceconv-bundle) &mdash; it is simpler, has
auto-discovery, and works on non-Perfetto trace formats.

The older `trace_processor util symbolize` and `trace_processor util
deobfuscate` subcommands
produce standalone symbol and deobfuscation files driven entirely by
environment variables, which must then be concatenated onto the trace by
hand.

#### Native symbolization

All tools (`trace_processor`, the `heap_profile` script)
honour the `PERFETTO_BINARY_PATH` environment variable:

```bash
PERFETTO_BINARY_PATH=somedir tools/heap_profile android --name ${NAME}
```

To produce a standalone symbol file for a trace you already collected:

```bash
PERFETTO_BINARY_PATH=somedir trace_processor util symbolize raw-trace > symbols
```

Alternatively, set `PERFETTO_SYMBOLIZER_MODE=index` and the symbolizer will
recursively index the directory for ELF files by Build ID, so filenames do not
need to match.

#### Java/Kotlin deobfuscation

Provide ProGuard/R8 maps via `PERFETTO_PROGUARD_MAP`, using the format
`packagename=map_filename[:packagename=map_filename...]`:

```bash
PERFETTO_PROGUARD_MAP=com.example.pkg1=foo.txt:com.example.pkg2=bar.txt \
  ./tools/heap_profile android -n com.example.app
```

To produce a standalone deobfuscation file for an existing trace:

```bash
PERFETTO_PROGUARD_MAP=com.example.pkg=proguard_map.txt \
  trace_processor util deobfuscate ${TRACE} > deobfuscation_map
```

#### Attaching the output to a trace

Both `symbols` and `deobfuscation_map` above are serialized `TracePacket`
protos, so for a **Perfetto protobuf trace** you can simply concatenate them:

```bash
cat ${TRACE} symbols > symbolized-trace
cat ${TRACE} deobfuscation_map > deobfuscated-trace
# or both:
cat ${TRACE} symbols deobfuscation_map > enriched-trace
```

The `tools/heap_profile` script does this automatically in its output directory
when `PERFETTO_BINARY_PATH` is set.

**Limitations:**

- The concatenation trick **only works for Perfetto protobuf traces**. Other
  trace formats (Chrome JSON, systrace, Firefox profile, etc.) cannot have
  `TracePacket` bytes appended this way. For those formats, use
  [Option 1](#option-1-traceconv-bundle) and load the symbols via
  `trace_processor_shell`.
- You must manage `PERFETTO_BINARY_PATH` / `PERFETTO_PROGUARD_MAP` by hand; none
  of the auto-discovery from Option 1 applies.

### Symbol lookup order

For each native mapping in the trace, the symbolizer looks for a file with
matching Build ID. For each search path `P`, it tries (in order):

1. Absolute path of the library file relative to `P`.
2. Same, with `base.apk!` stripped from the filename.
3. Basename of the library file relative to `P`.
4. Basename, with `base.apk!` stripped.
5. `P/.build-id/<first 2 hex digits>/<rest>.debug` (the standard
   [Fedora Build ID layout](https://fedoraproject.org/wiki/RolandMcGrath/BuildID#Find_files_by_build_ID)).

For example, `/system/lib/base.apk!foo.so` with build id `abcd1234...` is looked
up under a symbol path `P` at:

1. `P/system/lib/base.apk!foo.so`
2. `P/system/lib/foo.so`
3. `P/base.apk!foo.so`
4. `P/foo.so`
5. `P/.build-id/ab/cd1234...debug`

The first file with a matching Build ID wins. If the Build ID on disk differs
from the one recorded in the trace, the file is skipped.

### Using symbolization/deobfuscation from a C++ library

There is currently **no stable public C++ API** for performing symbolization or
deobfuscation in-process. The underlying implementation exists (`TraceToBundle`
in `src/traceconv/trace_to_bundle.h`, backed by `EnrichTrace` in
`src/trace_processor/util/trace_enrichment/trace_enrichment.h`), but it lives
under `src/` rather than `include/` and is not part of the public API surface.

If you need this, please +1 on
[GitHub issue #5534](https://github.com/google/perfetto/issues/5534) so we can
gauge demand and prioritise.

### Troubleshooting

`trace_processor bundle` always produces a bundle containing at least the
original trace. When it cannot add all the enrichment it wants, it prints a
summary of what is missing and how to fix it, then still exits successfully
&mdash; so check the output of the command even when it succeeds. It exits
non-zero only for genuine failures (unreadable input, unwritable output, or
an explicitly-provided `--proguard-map` that cannot be read).

Common messages and what they mean:

- **`N frames from M mappings: no usable symbols in the searched paths`**,
  followed by the mapping names: the tool searched the auto-discovered paths
  (plus any `--symbol-paths` you gave) but found no binary with a matching
  Build ID for those mappings, or only a stripped one. The `To fix this`
  block below it depends on where the binaries come from. If you build them
  yourself, point `--symbol-paths` at the unstripped build outputs. If they
  come from your OS, install its debug symbols: `apt install <package>-dbgsym`
  on Debian/Ubuntu (find the package with `dpkg -S <path>`), `dnf
  debuginfo-install <package>` on Fedora (`rpm -qf <path>`), both of which
  install to `/usr/lib/debug` where they are found automatically; on Android,
  the `symbols` directory of the matching platform build. Re-run with
  `--verbose` to see the Build IDs and every path that was tried.

- **`N frames from M mappings: kernel frames, no vmlinux in the searched
  paths`**: install the kernel debug package (`linux-image-$(uname -r)-dbg`
  on Debian/Ubuntu, `dnf debuginfo-install kernel` on Fedora) or point
  `--symbol-paths` at the `vmlinux` from your kernel build.

- **`N frames from M mappings: no build ID recorded, so symbols cannot be
  looked up`**: the trace's mappings have no Build ID, so symbols cannot be
  matched even with the right binaries. Rebuild the binaries with Build IDs
  (linker flag `-Wl,--build-id`) and re-record.

- **`N frames from M mappings: no backing file to read symbols from (JIT,
  anonymous or [vdso]-style mappings)`**: these frames come from memory with
  no binary behind it. Nothing offline can name them.

- **`Kernel function names: this trace contains function_graph events ...`**:
  the trace contains kernel addresses from `function_graph` (or similar
  ftrace events) recorded **without** `symbolize_ksyms`. These cannot be
  symbolized offline; re-record with `symbolize_ksyms: true`. See
  [Kernel ftrace events](#ftrace).

- **`no symbol paths were searched`**: automatic discovery was disabled
  (`--no-auto-symbol-paths`) and no explicit paths were given. Pass
  `--symbol-paths` with the directories to search.

- **`cannot create output file ...`**: the output path could not be created
  (e.g. the parent directory does not exist or is not writable). Check the
  path.

#### Could not find library

When symbolizing a profile with `--verbose` you may see messages like:

```text
  Could not symbolize 12 frames from 1 mapping:
    /data/app/invalid.app-wFgo3GRaod02wSvPZQ==/lib/arm64/somelib.so (12 frames)
      build ID: 44b7138abd5957b8d0a56ce86216d478
      no binary with a matching build ID in:
        /path/to/symbols/somelib.so (file not found)

  To fix this:
    If you build these binaries yourself, pass --symbol-paths DIR1,DIR2,... pointing at the unstripped build outputs.
    If they come from your OS, install its debug symbols; see https://perfetto.dev/docs/learning-more/symbolization
```

Check that `somelib.so` exists somewhere under one of the search paths
(`--symbol-paths` or an auto-discovered location). Then
compare the Build ID on disk to the one reported in the message using
`readelf -n /path/to/somelib.so`. If they do not match, the copy on disk is a
different build than the one on device and cannot be used.

Re-running `trace_processor bundle` with `--verbose` prints every path tried, which
usually makes it clear whether the file was missing entirely or found with the
wrong Build ID.

### Kernel frames in callstacks {#callstack-kernel-frames}

A sampled callstack can include **kernel** frames (e.g. perf sampling with
`callstack_sampling { kernel_frames: true }`). Unlike the userspace frames above,
these are symbolized **automatically on the device at record time** from
`/proc/kallsyms` &mdash; the offline tools in this section do not touch them.

For kernel frames to be named, the recording must be able to read
`/proc/kallsyms`, which requires running as root or lowering `kptr_restrict`:

```bash
echo 0 | sudo tee /proc/sys/kernel/kptr_restrict
```

If kernel frames show as hex addresses, this is a record-time permissions issue
and you have to re-record. This is the same KASLR constraint as for
[kernel ftrace events](#ftrace) below, but note the two use different
mechanisms: callstack kernel frames do **not** use the `symbolize_ksyms` ftrace
option &mdash; that flag only affects ftrace events.

## Kernel ftrace events: `symbolize_ksyms` {#ftrace}

If you are doing **system tracing** and seeing raw hexadecimal addresses where
you expected kernel function names &mdash; for example in
[function graph tracing](/docs/data-sources/funcgraph.md), in the
`blocked_function` field of an uninterruptible-sleep
[scheduling blockage](/docs/case-studies/scheduling-blockages.md), or in kprobe
events &mdash; the fix is **not** offline symbolization.

These kernel addresses are resolved **at record time** by enabling
`symbolize_ksyms` in the ftrace config:

```protobuf
data_sources: {
    config {
        name: "linux.ftrace"
        ftrace_config {
            symbolize_ksyms: true
            # ... your ftrace_events / function_graph config ...
        }
    }
}
```

This reads `/proc/kallsyms` on the device and embeds the (mangled) symbol map in
the trace. It requires that either `traced_probes` runs as root or
`kptr_restrict` has been lowered manually.

WARNING: `trace_processor bundle` and the offline symbolizers above **cannot** recover
kernel symbols. Perfetto deliberately does not store absolute kernel addresses
in the trace, because doing so would defeat
[KASLR](https://en.wikipedia.org/wiki/Address_space_layout_randomization) and
disclose the kernel memory layout. The symbol names are mangled on device so
this works without leaking absolute addresses. If you forgot to set
`symbolize_ksyms`, you have to re-record.

This flag applies only to ftrace **events**. Kernel frames captured inside
sampled callstacks are handled separately; see
[Kernel frames in callstacks](#callstack-kernel-frames).

## Userspace event names: atrace and ART method tracing {#userspace-event-names}

Some data sources record human-readable **name strings** rather than addresses
or stack frames. When those strings are obfuscated (e.g. an R8-obfuscated class
name), there is **no offline mechanism to deobfuscate them** &mdash; the name
must be emitted in a readable form at instrumentation time. This is distinct from
the Java/Kotlin **stack-frame** deobfuscation in
[the callstacks section](#callstacks), which applies only to heap dumps and
sampled callstacks.

This affects two cases today:

- **atrace / userspace slice names**: [atrace](/docs/data-sources/atrace.md)
  slice names (and other strings that ended up in a `TRACE_EVENT` literal) are
  recorded verbatim. There is no post-hoc mapping step.
- **ART method tracing**: the method names captured by ART method tracing are not
  run through the ProGuard/R8 deobfuscation path, so obfuscated builds will show
  obfuscated method names.

A `mapping.txt`-based deobfuscation path for these is in principle possible but
not currently implemented. Support is under discussion; see
[GitHub issue #6391](https://github.com/google/perfetto/issues/6391) for context
and to register interest.