This document is the spec, current implementation status, and handoff notes for the UI side of the Trace-To-Techno project. It describes the two-canvas graph-editor architecture, the state model, preset import semantics, the hard-earned gotchas, and the milestone plan.
For the TP/engine side, see trace-processor-design.md. For the conceptual background on modular synths, see background-on-synths.md.
Milestone 1 is complete. The two-canvas editor is functional with:
NodeGraph's built-in autoLayout API with real DOM-measured dimensions.Milestones 2 and 3 have concrete plans below; the plumbing they need (textproto round-trip via WASM, NodeGraphApi, BlockDescriptor registry) is already in place.
The SoundSynth plugin page is two node-graph canvases stacked vertically, with a left sidebar listing the trace's tracks and a bottom transport bar:
┌────────────────────────────────────────────────────────────────┐ │ Sound Synth · 256 presets loaded │ ├──────────────────┬─────────────────────────────────────────────┤ │ │ RACK · [+ Instrument] [+ Trace Source] │ │ │ │ │ Track Browser │ Trace sources, instrument nodes, master. │ │ (process tree) │ Macro wiring only. │ │ │ │ │ Click a track ├─────────────────────────────────────────────┤ │ to add it as a │ INSTRUMENT EDITOR · [+ Add Block] [▶ Test] │ │ rack trace │ │ │ source. │ Full internal patch of the selected │ │ │ instrument, with virtual INPUT/OUTPUT │ │ │ nodes and every SynthModule as its own │ │ │ node. Empty state when nothing selected. │ ├──────────────────┴─────────────────────────────────────────────┤ │ Transport: [▶ Render] [▶ Play / ⏹ Stop] │ └────────────────────────────────────────────────────────────────┘
SynthModule rendered as its own node. Hidden (empty state) when nothing is selected.Both canvases mutate the same underlying SynthesizeAudioArgs proto. Logically it is ONE graph — the two canvases are just different projections of the same flat module/wire list.
The entire UI state is a protos.ISynthesizeAudioArgs object. Every reactive bit of the UI renders directly from it. UI-only state (node positions, display names, mute/solo, the currently-edited-instrument id) is stashed into the opaque ui_state_json string fields of:
SynthPatch.ui_state_json — page-level stateSynthModule.ui_state_json — per-module stateThe JSON blobs are validated and typed by Zod schemas in patch_state.ts (PatchUiStateSchema, ModuleUiStateSchema).
Since the proto is flat, we use a naming convention to group modules into instruments:
instrumentId of the form inst_<base36>.${instrumentId}__${localName}.ui_state_json.nodeKind is instrument_root. In practice this is the preset‘s internal master mixer, renamed to ${instrumentId}__master on import. Its ui_state_json stores displayName, presetId, muted, soloed, level, x/y (rack position), outX/outY (position of the OUTPUT node inside the instrument editor — separate from x/y to avoid crosstalk between the two canvases), gateSource and freqSource (rack-level module IDs bound to the instrument’s virtual inputs).SynthWire entry.Inside an instrument, there is no proto module representing the “input from the rack”. Instead, wires that should receive the external signal use reserved virtual module IDs:
__input__gate — the gate/trigger signal from the rack__input__freq — the pitch CV from the rackThese IDs never appear on real modules. In the bottom canvas the UI synthesizes a virtual INPUT node at the top-left with output ports gate and freq; wires drawn from these ports are stored as real SynthWire entries with from_module: "__input__gate" (or __input__freq) and from_port: "out". The gate-vs-freq distinction is encoded in the virtual ID itself.
When a patch is actually sent to TP (either for rack rendering or for the Test button), buildRenderPatch() / buildTestPatch() walk all wires and rewrite virtual from_module IDs:
__input__gate → <instrument.gateSource>; the wire is dropped entirely if no source is bound. Same for __input__freq.__input__gate → the temporary TestPatternSource module ID (port out); __input__freq → the same test source (port freq).TP never sees the virtual IDs.
The instrument root (${instrumentId}__master) is a real Mixer module. It IS the “output” of the instrument. In the bottom canvas we render it with a distinctive style and label it “OUTPUT”. In the top canvas the same underlying module is rendered as the instrument node.
The two canvases must use different position fields for the same module, otherwise dragging OUTPUT in the bottom canvas teleports the instrument on the rack:
ui_state_json.x, .y → rack position (instrument node)ui_state_json.outX, .outY → instrument editor position (OUTPUT node)This separation is enforced in buildOutputNode() and the onNodeMove handler in instrument_canvas.ts, and in the auto-layout helper in patch_state.ts.
When the user picks a preset:
snakeToCamelDeep() (protobufjs expects camelCase).instrumentId.TestPatternSource module(s) — they exist only to make the preset self-contained for standalone rendering during preset generation. Record the stripped module's local ID (usually "arp").from_module that referenced a stripped test source:from_port: out → from_module: "__input__gate", from_port: outfrom_port: freq → from_module: "__input__freq", from_port: out${instrumentId}__.toObject/fromObject (NOT SynthModule.create, which does a shallow copy and can alias nested config).master mixer (now ${instrumentId}__master) as the instrument root via ui_state_json.nodeKind = "instrument_root" with display name, preset id, default level, etc.SynthPatch.${instrumentId}__master → master (rack master).layoutInstrumentModules() — a BFS-depth column layout that assigns initial positions. This is a rough pass; after the NodeGraph mounts the instrument canvas triggers the built-in autoLayout which measures actual DOM node sizes and lays them out properly.After import, the instrument is fully functional for the Test button. To hear it in the rack render, the user must wire a rack-level trace source into its gate (and optionally freq) input.
When the user clicks Test on an instrument:
SynthPatch containing:TestPatternSource (mode: ARPEGGIO, bpm: 128, bars: 4) — buildTestPatch() in patch_state.ts.__input__gate / __input__freq references rewritten to point at this test source’s out / freq ports.master Mixer receiving the instrument root's output.synthesizeAudio() with a small time window (16 × 1/48 seconds of trace time, stretched by the engine's 48× time dilation to 16 seconds of audio).The test patch is built on the fly and never persisted.
The Transport component enforces a single-stream invariant: at most one audio source plays at any time.
┌─ Transport state ─────────────────────────────┐ │ audioCtx: AudioContext | null │ │ sourceNode: AudioBufferSourceNode | null │ │ playing: boolean │ │ lastAutoPlayedBuf: ArrayBuffer | null │ │ playbackGeneration: number // monotonic │ └───────────────────────────────────────────────┘
Key invariants:
startPlayback() and stopPlayback() call increments playbackGeneration. An in-flight decodeAudioData await that resumes with gen !== playbackGeneration silently bails out. This prevents two concurrent Test clicks from both ending up starting a sound.stopPlayback() nulls onended before calling .stop(), so stale onended callbacks don't clear newer state. It also wraps .stop() and .disconnect() in try/catch (OK if already stopped).wavData transitions from non-null to null (new render begins), the Transport view calls stopPlayback() so the previous sound dies before the new one arrives.this.playing is true, even if wavData was cleared during a re-render, so the user always has a way to kill the current sound.All synth block types are described in a central registry (block_registry.ts):
interface BlockDescriptor { protoField: string; // e.g. "classic_osc" displayName: string; // e.g. "Classic Osc" description: string; category: 'source' | 'oscillator' | 'filter' | 'effect' | 'modulator' | 'utility'; hue: number; // Node color, 0-360 inputs: Array<{name: string; kind: PortKind}>; outputs: Array<{name: string; kind: PortKind}>; createDefault: () => protos.ISynthModule; renderParams: (mod: protos.ISynthModule, onChange: () => void) => m.Children; }
PortKind = 'audio' | 'cv' | 'gate' | 'freq' — informational only today, but can be used to color wires by signal type later.
Every block type in the proto has an entry. Milestone 1 ships full interactive panels for 8 blocks:
TestPatternSource, ClassicOsc, Adsr, MoogLadder, Svf, Waveshaper, Vca, MixerEvery other block has a generic read-only fallback panel built from proto field introspection, which Milestone 2 will replace with proper sliders/dropdowns.
Nodes:
TraceSliceSource at the rack level. Output port out. Inline controls for glob and signal type. Hue 140 (green). Right-click 3-dot menu → “Delete trace source”; or click-select + Delete key.instrument_root module. Input ports: gate, freq. Output port out. Body:in. Fixed position, cannot be removed.Connections on the rack:
out → Instrument gate / freq — stored in the instrument root's ui_state_json.gateSource / freqSource, NOT as a real SynthWire. The wire is synthesized by buildRenderPatch at render time.out → Master in — stored as a real SynthWire.Interactions:
ui_state_json.x/y).Only visible when an instrument is selected.
Nodes:
gate, freq. Not backed by a proto module; wires drawn from it use the reserved __input__gate / __input__freq IDs.BlockDescriptor. Inline parameter panels for the 8 “full panel” blocks, generic fallback for the rest. Click a node to select, press Delete (or use the 3-dot menu) to remove it.in. Real proto module. Its position is stored in ui_state_json.outX/outY (NOT x/y, which is reserved for the rack position of the same module).Interactions:
ui_state_json).SynthWire. Port names on each end are resolved via the BlockDescriptor.Auto-layout: when the instrument first mounts, the bottom canvas triggers NodeGraphApi.autoLayout() via an onReady callback, gated by a pendingAutoLayout flag. This uses real DOM node dimensions so the layout avoids overlaps. It's followed by recenter() to fit the graph into the canvas viewport.
The preset library is a single JSON file: test/data/music_synth_presets.json (256 presets, generated by tools/trace_to_techno/gen_presets.py).
Because the UI build walks ui/src/assets/ but skips symbolic links, the JSON file is duplicated to ui/src/assets/sound_synth/music_synth_presets.json. The Python generator writes both copies. There's a {r: /ui\/src\/assets\/(sound_synth\/(.*)[.]json)/, f: copyAssets} rule in ui/build.js that copies the JSON into the dist directory at build time.
If you modify gen_presets.py and regenerate, both copies are updated atomically.
preset_library.ts fetches the JSON once on page load via fetch(), runs snakeToCamelDeep() recursively on each preset's patch (only keys are rewritten; enum string values like "ARPEGGIO" are left alone), and then calls protos.SynthPatch.fromObject(camelPatch).
The result is exposed as a lazy, categorized, searchable PresetLibrary object with methods all(), byCategory(), categories(), search(query), and findByName(name).
preset_picker.ts is a Mithril modal-ish component overlaid on the main page. Category tabs along the top (All + per-category), full-text search box, scrollable list of entries with category indicator, description, and click-to-insert. Clicking an entry calls importPresetAsInstrument() and closes the picker.
If you're new to this codebase, save yourself the debugging hours by reading these before writing code:
Creating a plugin under ui/src/plugins/<name>/ registers it but does not enable it at startup. To make it load by default (so onTraceLoad fires and your page/sidebar entries appear), add the plugin ID to ui/src/core/embedder/default_plugins.ts. Otherwise users have to toggle it on via the Plugins page every time.
.pf-canvas width collapses inside flex containersThe NodeGraph widget renders as a plain block .pf-canvas div with height: 100% (when fillHeight: true is set). When this div is placed inside a display: flex parent, it behaves as a flex item and its width shrinks to content — which for an empty canvas is 0 — and the whole graph becomes invisible.
The fix is to wrap the NodeGraph in a position: relative parent with overflow: hidden, then an absolutely-positioned top/left/right/bottom: 0 inner div that contains the NodeGraph. This decouples the NodeGraph's size from flex layout.
See the .rack-canvas-wrapper / .rack-canvas-inner pattern (and the same for instrument canvas) in sound_synth_page.ts and instrument_canvas.ts.
Math.random() in view() for node positionsThis causes a redraw loop: each render passes a new random (x, y) to the NodeGraph → NodeGraph fires onNodeMove → you persist the new position → mithril redraws → new random again → forever.
Use ui.x ?? defaultX (NOT ui.x || defaultX — 0 is a valid position) for fallbacks, and ensure the fallback is deterministic.
The instrument's master mixer module is rendered in TWO canvases:
ui.x, ui.y)ui.outX, ui.outY)If you read/write the same field from both canvases, moving OUTPUT in the bottom canvas will teleport the instrument on the rack.
protos.SynthModule.create(m) does a shallow copy — the cloned message shares nested config objects with the original. If you then edit the clone‘s config (e.g. change the baseFreqHz of its classic_osc), you’ll also mutate the preset library entry.
Use fromObject(toObject(m)) for a true deep clone.
You generally don't need to call m.redraw() inside onclick, oninput, etc. Mithril batches a redraw after the handler returns. Explicit m.redraw() calls in those paths are redundant and can obscure control flow. Do call it from async callbacks (setTimeout, fetch, etc.) that fire outside the mithril event loop.
contextMenuItems is a header button, not right-clickThe NodeGraph widget renders a 3-dot “more_vert” button in each node's title bar that opens a PopupMenu with the attached contextMenuItems. It is not a real right-click handler. Users open it via left-click on that button. (This is actually fine; we also support Delete/Backspace on selected nodes.)
ui/src/assets/ must be real files, not symlinksThe dev server's scanDir / walk explicitly skips symbolic links (!stat.isSymbolicLink()). If you link from ui/src/assets/* into another directory, the file will never be copied into the dist and the UI will fail to fetch it. Either duplicate the file or teach the build script a per-file copy rule for that path.
After Milestone 1 the plugin looks like:
ui/src/plugins/dev.perfetto.SoundSynth/
index.ts # Plugin registration
sound_synth_page.ts # Top-level page: split layout, orchestration,
# Test button wiring, Render wiring
rack_canvas.ts # Top canvas (rack level)
instrument_canvas.ts # Bottom canvas (instrument internals)
block_registry.ts # BlockDescriptor[] for all 18 block types
patch_state.ts # Proto view, mutations, import/export,
# auto-layout helper
preset_library.ts # Fetch + parse music_synth_presets.json
preset_picker.ts # Preset picker modal
track_browser.ts # Left sidebar: process/track tree
transport.ts # Render + single-stream playback
Build integration:
ui/src/core/embedder/default_plugins.ts — plugin enabled by defaultui/build.js — {r: /ui\/src\/assets\/(sound_synth\/.*[.]json)/, f: copyAssets} rule for the preset JSONui/src/assets/sound_synth/music_synth_presets.json — real copy of the preset library (kept in sync with test/data/... by tools/trace_to_techno/gen_presets.py)__input__gate, __input__freq — virtual module IDs for the instrument INPUT node. Never used for real modules.__canvas_input__, __canvas_output__ — virtual canvas node IDs used only inside instrument_canvas.ts (not stored in the proto).inst_<base36>. Derived from Date.now() + counter.${instrumentId}__${localName}.master.Done:
patch_state.ts: instrument grouping, virtual INPUT, preset import, test patch builder, rack render patch builder, layout helper with auto-layout on import.block_registry.ts with descriptors for all 18 block types.TestPatternSource, ClassicOsc, Adsr, MoogLadder, Svf, Waveshaper, Vca, Mixer. Generic fallback for the rest.preset_library.ts: JSON fetch, snake→camel, TestPatternSource stripping, wire rewriting, categorized searchable API.preset_picker.ts: modal picker with category tabs and search.rack_canvas.ts: top canvas with trace source / instrument / master nodes, drag, wire, context menu delete, Delete key.instrument_canvas.ts: bottom canvas with virtual INPUT / real-module OUTPUT, inline param editing, block palette (+ Add Block), auto-layout via NodeGraphApi, independent OUTPUT position via outX/outY.sound_synth_page.ts: two-canvas split layout with proper flex / position:absolute chaining so .pf-canvas gets full width and height.transport.ts: single-stream audio playback with generation counter, auto-stop on render start, Stop button always visible while playing.Goal: every block fully editable with a well-designed param panel, and a more polished look/feel.
renderParams panels for the 10 remaining block types (all currently using the generic read-only fallback): NoiseOsc, FmOsc, PhaseDistortionOsc, FoldOsc, SyncOsc, SuperOsc, WavetableOsc, DrawbarOrgan, Lfo, Chorus, Delay. (Plus the legacy Vco and Envelope — bare minimum is fine there since they're only kept for back-compat.)PortKind metadata is already collected.ui_state_json.collapsed.toObject/fromObject.BlockDescriptor registry already has stubs for every block — just fill in renderParams with properly sized sliders and dropdowns. Look at the proto field definitions in protos/perfetto/trace_processor/synth.proto (enum values, numeric ranges in comments) for guidance. Reuse the slider() and dropdown() helpers at the top of block_registry.ts.audio / cv / gate / freq) are already stored but not visually distinguished. This is low-hanging fruit for color coding.ui_state_json.collapsed (bool, defaulting to false).node_panel.ts is structured — it’s a good model.view(). Use ??, not ||. See Gotcha #3.Goal: turn the editor into a real playground with file sync, custom presets, trace-aware rhythm, and undo/redo.
synthArgsToText / synthArgsToPb in ui/src/base/proto_utils_wasm.ts). Build a FileSyncManager:FileSystemFileHandle via the File System Access API.AsyncGuard from ui/src/base/async_guard.ts).localStorage. These appear alongside built-in presets in the picker (maybe with a 👤 badge).SynthesizeAudioArgs proto on every significant mutation, stack up to ~50 snapshots. Revert by swapping the active snapshot.src/base/async_guard.ts for the debounce primitive. The canonical conflict-resolution UX is: show a modal with three buttons and two “Download” links so the user can diff externally.localStorage.getItem('soundsynth_user_presets') storing a JSON array in the same shape as the built-in preset file. Use preset_library.ts as the model for parsing.protos.SynthesizeAudioArgs.encode(state).finish() then .decode(bytes) for a guaranteed-independent copy. That's slower than toObject/fromObject but the proto is small (~KB).