docs: add PerfettoSQL syntax page and reorg others
This CL adds a page for PerfettoSQL syntax and reorganizes other
pages and cleanup unnecessary documentation while we're here.
Change-Id: Idad2faffaae7f0dc0be70c8076a18c6dc14937b3
diff --git a/docs/analysis/builtin.md b/docs/analysis/builtin.md
index 29dd157..84deaab 100644
--- a/docs/analysis/builtin.md
+++ b/docs/analysis/builtin.md
@@ -1,4 +1,4 @@
-# Built-in Functions
+# PerfettoSQL Built-ins
These are functions built into C++ which reduce the amount of boilerplate which
needs to be written in SQL.
diff --git a/docs/analysis/common-queries.md b/docs/analysis/common-queries.md
index 018ab94..adae686 100644
--- a/docs/analysis/common-queries.md
+++ b/docs/analysis/common-queries.md
@@ -1,4 +1,4 @@
-# Common queries
+# PerfettoSQL Common Queries
This page acts as a reference guide for queries which often appear when
performing ad-hoc analysis.
diff --git a/docs/analysis/perfetto-sql-syntax.md b/docs/analysis/perfetto-sql-syntax.md
new file mode 100644
index 0000000..6b0cb2c
--- /dev/null
+++ b/docs/analysis/perfetto-sql-syntax.md
@@ -0,0 +1,103 @@
+# PerfettoSQL Syntax
+*This page documents the syntax of PerfettoSQL, a dialect of SQL used in trace
+processor and other Perfetto analysis tools to query traces.*
+
+PerfettoSQL is a direct descendent of the
+[dialect of SQL implemented by SQLite](https://www.sqlite.org/lang.html).
+Specifically, any SQL valid in SQLite is also valid in PerfettoSQL.
+
+Unfortunately, the SQLite syntax alone is not sufficient for two reasons:
+1. It is quite basic e.g. it does not support creating functions or macros
+2. It cannot be used to access features which are only available in Perfetto
+tooling e.g. it cannot be used to create efficient analytic tables, import
+modules from the PerfettoSQL standard library etc.
+
+For this reason, PerfettoSQL adds new pieces of syntax which make the experience
+of writing SQL queries better. All such additons include the keyword `PERFETTO`
+to make it clear that they are PerfettoSQL-only.
+
+<!-- TODO(b/290185551): we should really talk about our "recommendations" (e.g.
+using CREATE PERFETTO TABLE instead of CREATE TABLE) somewhere and reference it
+here. -->
+
+## Including PerfettoSQL modules
+`INCLUDE PERFETTO MODULE` is used to import all tables/views/functions/macros
+defined in a PerfettoSQL module (e.g. from the
+[PerfettoSQL standard library](/docs/analysis/stdlib-docs.autogen)).
+
+Note that this statement acts more similar to `#include` statements in C++
+rather than `import` statements from Java/Python. Specifically, all objects
+in the module become available in the global namespace without being qualified
+by the module name.
+
+Example:
+```sql
+-- Include all tables/views/functions from the android.startup.startups module
+-- in the standard library.
+INCLUDE PERFETTO MODULE android.startup.startups;
+
+-- Use the android_startups table defined in the android.startup.startups
+-- module.
+SELECT *
+FROM android_startups;
+```
+
+## Defining functions
+`CREATE PEFETTO FUNCTION` allows functions to be defined in SQL. The syntax is
+similar to the syntax in PostgreSQL or GoogleSQL.
+
+<!-- TODO(b/290185551): talk about different possible argument/return types. -->
+
+Example:
+```sql
+-- Create a scalar function with no arguments.
+CREATE PERFETTO FUNCTION constant_fn() RETURNS INT AS SELECT 1;
+
+-- Create a scalar function taking two arguments.
+CREATE PERFETTO FUNCTION add(x INT, y INT) RETURNS INT AS SELECT $x + $y;
+
+-- Create a table function with no arguments
+CREATE PERFETTO FUNCTION constant_tab_fn()
+RETURNS TABLE(ts LONG, dur LONG) AS
+SELECT column1 as ts, column2 as dur
+FROM (
+ VALUES
+ (100, 10),
+ (200, 20)
+);
+
+-- Create a table function with one argument
+CREATE PERFETTO FUNCTION sched_by_utid(utid INT)
+RETURNS TABLE(ts LONG, dur LONG, utid INT) AS
+SELECT ts, dur, utid
+FROM sched
+WHERE utid = $utid;
+```
+
+## Creating efficient tables
+`CREATE PERFETTO TABLE` allows defining tables optimized for analytic queries
+on traces. These tables are both more performant and more memory efficient than
+SQLite native tables created with `CREATE TABLE`.
+
+Note however the full feature set of `CREATE TABLE` is not supported:
+1. Perfetto tables cannot be inserted into and are read-only after creation
+2. Perfetto tables must be defined and populated using a `SELECT` statement.
+ They cannot be defined by column names and types.
+
+Example:
+```sql
+-- Create a Perfetto table with constant values.
+CREATE PERFETTO TABLE constant_table AS
+SELECT column1 as ts, column2 as dur
+FROM (
+ VALUES
+ (100, 10),
+ (200, 20)
+);
+
+-- Create a Perfetto table with a query on another table.
+CREATE PERFETTO TABLE slice_sub_table AS
+SELECT *
+FROM slice
+WHERE name = 'foo';
+```
diff --git a/docs/analysis/trace-processor.md b/docs/analysis/trace-processor.md
index f1ef185..8b5fdc2 100644
--- a/docs/analysis/trace-processor.md
+++ b/docs/analysis/trace-processor.md
@@ -509,24 +509,6 @@
The metrics subsystem is a significant part of trace processor and thus is
documented on its own [page](/docs/analysis/metrics.md).
-## Annotations
-
-TIP: To see how to add to add a new annotation to trace processor, see the
-checklist [here](/docs/contributing/common-tasks.md#new-annotation).
-
-Annotations attach a human-readable description to a slice in the trace. This
-can include information like the source of a slice, why a slice is important and
-links to documentation where the viewer can learn more about the slice.
-In essence, descriptions act as if an expert was telling the user what the slice
-means.
-
-For example, consider the `inflate` slice which occurs during view inflation in
-Android. We can add the following description and link:
-
-**Description**: Constructing a View hierarchy from pre-processed XML via
-LayoutInflater#layout. This includes constructing all of the View objects in the
-hierarchy, and applying styled attributes.
-
## Creating derived events
TIP: To see how to add to add a new annotation to trace processor, see the
@@ -557,23 +539,6 @@
The other benefit of aligning the two is that changes in metrics are
automatically kept in sync with what the user sees in the UI.
-## Alerts
-
-Alerts are used to draw the attention of the user to interesting parts of the
-trace; this are usually warnings or errors about anomalies which occurred in the
-trace.
-
-Currently, alerts are not implemented in the trace processor but the API to
-create derived events was designed with them in mind. We plan on adding another
-column `alert_type` (name to be finalized) to the annotations table which can
-have the value `warning`, `error` or `null`. Depending on this value, the
-Perfetto UI will flag these events to the user.
-
-NOTE: we do not plan on supporting case where alerts need to be added to
- existing events. Instead, new events should be created using annotations
- and alerts added on these instead; this is because the trace processor
- storage is monotonic-append-only.
-
## Python API
The trace processor Python API is built on the existing HTTP interface of `trace processor`
diff --git a/docs/toc.md b/docs/toc.md
index b0d8fc1..0ed6a88 100644
--- a/docs/toc.md
+++ b/docs/toc.md
@@ -1,6 +1,8 @@
-* [Overview](README.md)
+* [Introduction](README.md)
-* [Tracing 101](tracing-101.md)
+* [Overview](#)
+ * [Tracing 101](tracing-101.md)
+ * [FAQ](faq.md)
* [Quickstart](#)
* [Record traces on Android](quickstart/android-tracing.md)
@@ -11,10 +13,8 @@
* [Heap profiling](quickstart/heap-profiling.md)
* [Callstack sampling on Android](quickstart/callstack-sampling.md)
-* [FAQ](faq.md)
-
* [Case studies](#)
- * [Android boot tracing](case-studies/android-boot-tracing.md)
+ * [Tracing Android boot](case-studies/android-boot-tracing.md)
* [Debugging memory usage](case-studies/memory.md)
* [Data sources](#)
@@ -40,13 +40,14 @@
* [Interceptors](instrumentation/interceptors.md)
* [Trace analysis](#)
- * [Trace Processor (SQL)](analysis/trace-processor.md)
- * [Batch Trace Processor](analysis/batch-trace-processor.md)
- * [Standard library](analysis/stdlib-docs.autogen)
- * [Built-in Functions](analysis/builtin.md)
+ * [Trace Processor](analysis/trace-processor.md)
+ * [PerfettoSQL Syntax](analysis/perfetto-sql-syntax.md)
+ * [PerfettoSQL Standard Library](analysis/stdlib-docs.autogen)
+ * [PerfettoSQL Tables](analysis/sql-tables.autogen)
+ * [PerfettoSQL Built-ins](analysis/builtin.md)
+ * [PerfettoSQL Common Queries](analysis/common-queries.md)
* [Trace-based metrics](analysis/metrics.md)
- * [Common queries](analysis/common-queries.md)
- * [SQL tables](analysis/sql-tables.autogen)
+ * [Batch Trace Processor](analysis/batch-trace-processor.md)
* [Stats table](analysis/sql-stats.autogen)
* [Pivot tables](analysis/pivot-tables.md)
diff --git a/infra/perfetto.dev/src/gen_sql_tables_reference.js b/infra/perfetto.dev/src/gen_sql_tables_reference.js
index 2f77cea..72800b5 100644
--- a/infra/perfetto.dev/src/gen_sql_tables_reference.js
+++ b/infra/perfetto.dev/src/gen_sql_tables_reference.js
@@ -325,7 +325,8 @@
graph += '\n```\n';
}
- let md = graph;
+ let title = '# PerfettoSQL Tables\n'
+ let md = title + graph;
for (const tableGroup of tableGroups) {
md += `## ${tableGroup}\n`
for (const table of tablesByGroup[tableGroup]) {
diff --git a/infra/perfetto.dev/src/gen_stdlib_docs_md.py b/infra/perfetto.dev/src/gen_stdlib_docs_md.py
index fe61806..7959053 100644
--- a/infra/perfetto.dev/src/gen_stdlib_docs_md.py
+++ b/infra/perfetto.dev/src/gen_stdlib_docs_md.py
@@ -32,11 +32,11 @@
self.files_md = [
FileMd(module_name, file_dict) for file_dict in module_files
]
- self.summary_objs = "\n".join(
+ self.summary_objs = '\n'.join(
file.summary_objs for file in self.files_md if file.summary_objs)
- self.summary_funs = "\n".join(
+ self.summary_funs = '\n'.join(
file.summary_funs for file in self.files_md if file.summary_funs)
- self.summary_view_funs = "\n".join(file.summary_view_funs
+ self.summary_view_funs = '\n'.join(file.summary_view_funs
for file in self.files_md
if file.summary_view_funs)
@@ -48,13 +48,13 @@
long_s.append(f'### {file.import_key}')
if file.objs:
long_s.append('#### Views/Tables')
- long_s.append("\n".join(file.objs))
+ long_s.append('\n'.join(file.objs))
if file.funs:
long_s.append('#### Functions')
- long_s.append("\n".join(file.funs))
+ long_s.append('\n'.join(file.funs))
if file.view_funs:
- long_s.append('#### View Functions')
- long_s.append("\n".join(file.view_funs))
+ long_s.append('#### Table Functions')
+ long_s.append('\n'.join(file.view_funs))
return '\n'.join(long_s)
@@ -70,75 +70,76 @@
# Add imports if in file.
for data in file_dict['imports']:
# Anchor
- anchor = f'obj/{module_name}/{data["name"]}'
+ anchor = f'''obj/{module_name}/{data['name']}'''
# Add summary of imported view/table
- desc = data["desc"].split(".")[0]
- summary_objs_list.append(f'[{data["name"]}](#{anchor})|'
- f'{file_dict["import_key"]}|'
- f'{desc}')
+ desc = data['desc'].split('.')[0]
+ summary_objs_list.append(f'''[{data['name']}](#{anchor})|'''
+ f'''{file_dict['import_key']}|'''
+ f'''{desc}''')
- self.objs.append(f'\n\n<a name="{anchor}"></a>'
- f'**{data["name"]}**, {data["type"]}\n\n'
- f'{data["desc"]}\n')
+ self.objs.append(f'''\n\n<a name="{anchor}"></a>'''
+ f'''**{data['name']}**, {data['type']}\n\n'''
+ f'''{data['desc']}\n''')
self.objs.append('Column | Description\n------ | -----------')
for name, desc in data['cols'].items():
self.objs.append(f'{name} | {desc}')
- self.objs.append("\n\n")
+ self.objs.append('\n\n')
# Add functions if in file
for data in file_dict['functions']:
# Anchor
- anchor = f'fun/{module_name}/{data["name"]}'
+ anchor = f'''fun/{module_name}/{data['name']}'''
# Add summary of imported function
- summary_funs_list.append(f'[{data["name"]}](#{anchor})|'
- f'{file_dict["import_key"]}|'
- f'{data["return_type"]}|'
- f'{data["desc"].split(".")[0]}')
+ summary_funs_list.append(f'''[{data['name']}](#{anchor})|'''
+ f'''{file_dict['import_key']}|'''
+ f'''{data['return_type']}|'''
+ f'''{data['desc'].split('.')[0]}''')
self.funs.append(
- f'\n\n<a name="{anchor}"></a>'
- f'**{data["name"]}**\n'
- f'{data["desc"]}\n\n'
- f'Returns: {data["return_type"]}, {data["return_desc"]}\n\n')
+ f'''\n\n<a name="{anchor}"></a>'''
+ f'''**{data['name']}**\n'''
+ f'''{data['desc']}\n\n'''
+ f'''Returns: {data['return_type']}, {data['return_desc']}\n\n''')
if data['args']:
self.funs.append('Argument | Type | Description\n'
'-------- | ---- | -----------')
for name, arg_dict in data['args'].items():
- self.funs.append(f'{name} | {arg_dict["type"]} | {arg_dict["desc"]}')
+ self.funs.append(
+ f'''{name} | {arg_dict['type']} | {arg_dict['desc']}''')
- self.funs.append("\n\n")
+ self.funs.append('\n\n')
- # Add view functions if in file
+ # Add table functions if in file
for data in file_dict['view_functions']:
# Anchor
- anchor = rf'view_fun/{module_name}/{data["name"]}'
+ anchor = rf'''view_fun/{module_name}/{data['name']}'''
# Add summary of imported view function
- summary_view_funs_list.append(f'[{data["name"]}](#{anchor})|'
- f'{file_dict["import_key"]}|'
- f'{data["desc"].split(".")[0]}')
+ summary_view_funs_list.append(f'''[{data['name']}](#{anchor})|'''
+ f'''{file_dict['import_key']}|'''
+ f'''{data['desc'].split('.')[0]}''')
- self.view_funs.append(f'\n\n<a name="{anchor}"></a>'
- f'**{data["name"]}**\n'
- f'{data["desc"]}\n\n')
+ self.view_funs.append(f'''\n\n<a name="{anchor}"></a>'''
+ f'''**{data['name']}**\n'''
+ f'''{data['desc']}\n\n''')
if data['args']:
self.view_funs.append('Argument | Type | Description\n'
'-------- | ---- | -----------')
for name, arg_dict in data['args'].items():
self.view_funs.append(
- f'{name} | {arg_dict["type"]} | {arg_dict["desc"]}')
+ f'''{name} | {arg_dict['type']} | {arg_dict['desc']}''')
self.view_funs.append('\n')
self.view_funs.append('Column | Description\n' '------ | -----------')
for name, desc in data['cols'].items():
self.view_funs.append(f'{name} | {desc}')
- self.view_funs.append("\n\n")
+ self.view_funs.append('\n\n')
- self.summary_objs = "\n".join(summary_objs_list)
- self.summary_funs = "\n".join(summary_funs_list)
- self.summary_view_funs = "\n".join(summary_view_funs_list)
+ self.summary_objs = '\n'.join(summary_objs_list)
+ self.summary_funs = '\n'.join(summary_funs_list)
+ self.summary_view_funs = '\n'.join(summary_view_funs_list)
def main():
@@ -158,10 +159,44 @@
common_module = modules_dict.pop('common')
with open(args.output, 'w') as f:
- f.write("# SQL standard library\n"
- "To import any function, view_function, view or table simply run "
- "`INCLUDE PERFETTO MODULE {import key};` in your SQL query.\n"
- "## Summary\n")
+ f.write('''
+# PerfettoSQL standard library
+*This page documents the PerfettoSQL standard library.*
+
+## Introduction
+The PerfettoSQL standard library is a repository of tables, views, functions
+and macros, contributed by domain experts, which make querying traces easier
+Its design is heavily inspired by standard libraries in languages like Python,
+C++ and Java.
+
+Some of the purposes of the standard library include:
+1) Acting as a way of sharing and commonly written queries without needing
+to copy/paste large amounts of SQL.
+2) Raising the abstraction level when exposing data in the trace. Many
+modules in the standard library convert low-level trace concepts
+e.g. slices, tracks and into concepts developers may be more familar with
+e.g. for Android developers: app startups, binder transactions etc.
+
+Standard library modules can be included as follows:
+```
+-- Include all tables/views/functions from the android.startup.startups
+-- module in the standard library.
+INCLUDE PERFETTO MODULE android.startup.startups;
+
+-- Use the android_startups table defined in the android.startup.startups
+-- module.
+SELECT *
+FROM android_startups;
+```
+
+More information on importing modules is available in the
+[syntax documentation](/docs/analysis/perfetto-sql-syntax#including-perfettosql-modules)
+for the `INCLUDE PERFETTO MODULE` statement.
+
+<!-- TODO(b/290185551): talk about experimental module and contributions. -->
+
+## Summary
+''')
summary_objs = [common_module.summary_objs
] if common_module.summary_objs else []
@@ -191,21 +226,21 @@
'Name | Import | Description\n'
'---- | ------ | -----------\n')
f.write('\n'.join(summary_objs))
- f.write("\n")
+ f.write('\n')
if summary_funs:
- f.write("### Functions\n\n"
+ f.write('### Functions\n\n'
'Name | Import | Return type | Description\n'
'---- | ------ | ----------- | -----------\n')
f.write('\n'.join(summary_funs))
- f.write("\n")
+ f.write('\n')
if summary_view_funs:
- f.write("### View Functions\n\n"
+ f.write('### Table Functions\n\n'
'Name | Import | Description\n'
'---- | ------ | -----------\n')
f.write('\n'.join(summary_view_funs))
- f.write("\n")
+ f.write('\n')
f.write('\n\n')
f.write(common_module.print_description())
