Plastic Labs Internal Wiki

Recently Updated

cloudevents-full-tracing

16 min read

CloudEvents Full-Fidelity Tracing

Status: Draft (eng-reviewed 2026-06-16) | Owner: vineeth Related: reasoning-traces.md (superseded for storage — see §0), telemetry-analytics.md, done/llm-client-refactor.md

0. What changed in eng review (2026-06-16)

This spec was reviewed and the architecture locked. Headline decisions:

  • Storage substrate is Parquet; query engine is swappable (DuckDB locally, DuckDB/ClickHouse in cloud via Xatu). Not a live DuckDB database file.
  • Trace bulk lives in the event stream, never in Honcho’s Postgres. reasoning-traces.md’s Postgres payload store is superseded; its provenance + agent-introspection ideas become a deferred follow-on that stores a payload-free row pointing at the event store.
  • Self-contained trace stream — no claimed join to the sampled metrics event. The two streams serve different masters (billing vs audit).
  • Additive now, explicit sunset later — the JSONL REASONING_TRACES_FILE path and Langfuse get a concrete retirement roadmap (§10), not “deferred forever.”
  • Core-only scope — this spec is emit + correlation + offline local importer. Excadrill/eevee reuse, the Groudon dashboard, and autoresearch loops are named follow-ons (§13).

1. Problem Statement

Honcho already emits a structured CloudEvents telemetry stream (src/telemetry/): a buffered, batched, retried HTTP emitter (emitter.py) POSTs application/cloudevents+json to a configurable endpoint, with deterministic idempotency IDs and namespace-based tenant routing. Events are well-modeled Pydantic classes (llm.call.completed, agent.iteration, dialectic.completed, dream.run, …).

But these events are accounting events — token counts, model ids, durations, finish reasons — and deliberately carry no payload. The actual prompt / completion / context-window content lives in a separate path: the REASONING_TRACES_FILE JSONL writer (src/telemetry/reasoning_traces.py), consumed downstream by minccino for training data.

Three gaps:

  1. No context-window audit. We cannot ask “what was the exact context window fed to the model at iteration N of run R?” from the event stream.
  2. Two disjoint trace systems. Metrics (CloudEvents) and content (JSONL) are separate, with no shared correlation. Reconstructing a full agentic run requires manual joining.
  3. No local, owned trace store. Auditing leans on Langfuse (hosted) or hand-reading JSONL. No self-hosted, queryable system of record.

2. Goals / Non-Goals

Goals

  • G1: Capture a faithful, replay-grade reconstruction of each LLM call as Honcho issues it — the normalized messages array at the executor boundary, plus structured output including thinking blocks and provider signatures — correlated into the trace model. (Reworded from “exact provider wire-bytes”: we capture Honcho’s normalized call at the executor, not the post-backend wire payload. The backend adapters transform further; that boundary is documented, not hidden.)
  • G2: Make every LLM path reconstructable — agentic loops, single-shot deriver/summarizer/dialectic/dream calls, and streamed calls (captured at stream finalization; see §7).
  • G3: Design correlation as a span tree so future parallel subagents and dreamer forking are traceable without rework.
  • G4: Ship a local offline importer (CloudEvents → Parquet) so traces are queryable from docker-compose with zero external dependencies, using the same Parquet schema Xatu produces in cloud.
  • G5: Be additive on landing, with an explicit sunset roadmap (§10). Payload capture is default-off; the existing metrics events are untouched. The JSONL path and Langfuse get concrete retirement criteria rather than living forever.

Non-Goals

  • NG1 (now scheduled, not “never”): Superseding the JSONL path is not done in this spec but has a concrete trigger in §10 (replay-grade parity → retire JSONL).
  • NG2 (now scheduled): Langfuse demotion criteria are in §10.
  • NG3: Building a production-grade hosted trace backend. Cloud retention reuses Xatu.
  • NG4: A polished trace-viewer UI.
  • NG5: Storing payloads in Honcho’s Postgres (explicitly rejected — see §11).
  • NG6: A live local collector HTTP service (replaced by the offline importer — see §9).

3. Background: what already exists

  • Emitter (emitter.py:864): deque(maxlen=max_buffer_size) that drops oldest on overflow (and records a buffer_full Prometheus drop — prometheus/metrics.py:120), exponential-backoff retry, graceful-shutdown flush. Singleton per process.
  • Idempotency (events/base.py:17): generate_event_id() folds event_type + resource_id + timestamp + honcho_version → deterministic evt_* id. Note: because timestamp is folded in, two emissions of byte-identical content at different times get different event ids — see §6.2.
  • Correlation (events/llm.py:137, events/agent.py): run_id (nanoid) + iteration. tool_call_seq (events/agent.py:288) indexes tool executions after a model response, not LLM calls — it is not a valid LLM-call join field.
  • Content in hand at emit site: executor.py — the messages array is in scope in honcho_llm_call_inner (param at :280, used at :338/:394), where _emit_llm_call_completed is called (:347/:371/:407). reasoning_traces.log_reasoning_trace already receives prompt + messages + response.
  • Sampler (emitter.py:30): _should_sample() keyed on run_id; ground_truth events bypass it. In practice Plastic Labs runs at rate 1.0 — sampling is effectively unused, so cross-stream divergence is not an operational concern (it still informs the self-contained-stream decision in §5).

4. Correlation model: span tree

Flat run_id + iteration cannot express forked/parallel work. Adopt the OpenTelemetry data model (model only, not transport):

FieldMeaning
trace_idthe whole top-level operation; stable across all forks
span_idone unit of work (an agent / subagent / fork invocation)
parent_span_idthe span that spawned this one; null at the root
iterationloop iteration within a span (1-indexed)
attemptretry/fallback attempt within an iteration (disambiguates retries)
step_seqmonotonic counter within a span for total ordering of its steps
  • run_id today becomes span_id. A new trace_id is minted at the top-level entrypoint and threaded down; forks inherit trace_id, get a fresh span_id, set parent_span_id.
  • step_seq is threaded explicitly on LLMTelemetryContext (alongside run_id/ iteration/span_id), not a contextvar. Rationale: the telemetry layer already threads correlation explicitly to avoid the per-iteration reset that iteration_scope() applies to the contextvars in src/utils/types.py; a step_seq contextvar would reset mid-span and corrupt ordering. The span entrypoint owns the counter’s lifetime.
  • attempt is carried so a retry/fallback chain (multiple provider calls for the same (span_id, iteration)) is unambiguous within the trace stream itself.
  • No global total order across parallel spans — concurrent branches are concurrent; the trace is a tree/DAG ordered per-branch.
  • Even before forking ships, every entrypoint mints trace_id == span_id (root span), so single-call paths (deriver/summarizer/dialectic/dream) are reconstructable (G2).
trace_id  (top-level operation)
 ├─ span A (run)                       parent_span_id = null
 │   ├─ iteration 1
 │   │   └─ llm.call.traced  step_seq=1  attempt=1  input_message_refs=[...]
 │   ├─ iteration 1  (retry)
 │   │   └─ llm.call.traced  step_seq=2  attempt=2   ← retry disambiguated by attempt
 │   └─ iteration 2
 │       └─ llm.call.traced  step_seq=3  attempt=1
 └─ span B (forked subagent)           parent_span_id = A     ← future
     └─ iteration 1
         └─ llm.call.traced  step_seq=1  (shares A's context prefix by hash)

5. New events

Two new ground-truth event types. The existing metrics events are untouched (G5). The trace stream is self-contained — it does not claim a join to llm.call.completed. It carries its own token copy, so cost is computable from the trace alone, and the two streams (billing vs audit) are intentionally decoupled.

5.1 llm.call.traced

class LLMCallTracedEvent(BaseEvent):
    _event_type   = "llm.call.traced"
    _schema_version = 1
    _category     = "llm"
    _volume_class = "ground_truth"          # never sampled — system of record
 
    # span-tree correlation (self-contained; no join to llm.call.completed)
    trace_id: str
    span_id: str
    parent_span_id: str | None = None
    iteration: int | None = None
    step_seq: int
    attempt: int = 1                         # disambiguates retries within an iteration
    was_fallback: bool = False               # provider fallback on this attempt
    parent_event_id: str | None = None       # chain to prior step's event id
 
    # path identity
    call_purpose: CallPurpose | None = None
    parent_category: str | None = None        # representation|dialectic|dream|summary
    transport: ModelTransport
    provider_label: str | None = None         # vendor behind a relay (matches metrics event)
    model: str
 
    # THE CONTEXT WINDOW (content-addressed; see §6)
    input_message_refs: list[str]             # ordered content hashes
    system_prompt_ref: str | None = None
    tool_schema_refs: list[str] | None = None # content-addressed FULL tool schemas (not names)
    tool_choice: str | dict | None = None
 
    # output — replay-grade (see §6.4)
    output_content_ref: str | None = None
    output_tool_calls: list[dict] | None = None     # incl. thought_signature where present
    output_thinking_ref: str | None = None          # thinking blocks (Anthropic) / reasoning
    output_signatures: dict | None = None            # signed thinking/tool material for replay
    raw_response_ref: str | None = None              # escape hatch: content-addressed raw provider response
    finish_reason: str | None = None
 
    # cheap accounting copy for self-contained cost queries
    provider_input_tokens: int = 0
    provider_output_tokens: int = 0
    cache_read_tokens: int = 0
    cache_creation_tokens: int = 0
    was_truncated: bool = False
 
    def get_resource_id(self) -> str:
        # NOTE: tool_call_seq removed — it indexed tool executions, not LLM calls.
        return f"{self.span_id}:{self.iteration}:{self.attempt}:{self.step_seq}"

5.2 trace.content

A single unique message, emitted once per run and referenced by hash.

class TraceContentEvent(BaseEvent):
    _event_type   = "trace.content"
    _schema_version = 1
    _category     = "llm"
    _volume_class = "ground_truth"
 
    # hash covers the FULL message identity, not just text — role/tool_call_id are
    # INSIDE the hash so identical text under different roles never collides.
    content_hash: str            # sha256(canonical({role, content, tool_call_id})) — global
    role: str                    # system|user|assistant|tool
    content: dict | str          # structured message (blocks/tool results), not flattened text
    tool_call_id: str | None = None
 
    def get_resource_id(self) -> str:
        return self.content_hash
 
    def generate_id(self) -> str:
        # Override base: derive the CloudEvent id from content_hash with NO timestamp,
        # so accidental re-sends of identical content dedupe at transport too (the base
        # impl folds timestamp+version → identical content would otherwise get new ids).
        return f"evt_content_{self.content_hash[:32]}"

Tenant identity is not a field on the event — it rides on the CloudEvent source (/honcho/{namespace}/{category}). Xatu enriches each event downstream, deriving tenant_id from the namespace; access control runs against the enriched tenant_id (see §6.3).

6. Content-addressing

6.1 Why (O(N²) → O(N), storage and bandwidth)

In an agentic loop, iteration N’s context window ⊇ iteration N−1’s (append-only). Capturing input_messages in full every iteration is O(N²) bytes. Content-addressing emits each unique message once and references it by hash → per-run payload linear in unique messages.

6.2 Making the O(N) real on the wire (emit-once + content-addressed id)

The collector dedupes content at storage via the content_hash primary key. But that alone does not make bandwidth O(N): the base generate_event_id() folds the timestamp (base.py:42), so re-emitting the same message at each iteration would ship N copies over the wire. Two mechanisms fix this:

  1. Per-run emitted-hash set — a trace.content for a hash already emitted this run is skipped. Each unique message ships once per run. (Folds with the hash memoization in §6.5.)
  2. Content-addressed event id — TraceContentEvent.generate_id() is overridden to derive from content_hash (no timestamp), so any accidental cross-process/cross-retry re-send also dedupes at transport.

6.3 Tenant identity (via Xatu enrichment) and the access invariant

Dedup is global by content hash: content_hash = sha256(canonical({role, content, tool_call_id})). The content store has no tenant column. Tenant attribution is added by Xatu enrichment (namespace → tenant_id) on the event stream.

Access invariant (binding on all multi-tenant read paths, incl. the Groudon dashboard follow-on): content is resolvable only by joining from events the requesting tenant_id owns. There is no standalone content-by-hash read endpoint. Every query filters by tenant_id on the (enriched) events and reaches content through those scoped events only. Honcho-authored content (system_prompt_ref, scaffold) is tagged so tenant-facing views can withhold it by default (internal audit sees all).

Per-namespace hard deletion over globally-shared rows requires reference accounting at the namespace/Xatu layer (tracked OQ3). Not blocking for capture.

6.4 Replay-grade content

content is the structured message Honcho holds (multi-part blocks, tool results, cache hints), not flattened text. Output capture includes thinking blocks, thought_signature / signed thinking material, reasoning details, and a content-addressed raw_response_ref escape hatch — the fields the LLM-client refactor deliberately preserved for replay. This is the fidelity level minccino/replay needs and the prerequisite for retiring the JSONL path (§10).

6.5 Hashing cost is O(N), not O(N²)

Hash each message once per run (memoize message → hash; the append-only structure means only messages appended since the previous iteration need hashing). This keeps per-run sha256 work linear, matching the storage win — without it, re-hashing the growing prefix every iteration is O(N²) CPU on the synchronous LLM path.

7. Emit-site changes

  • executor.py (inside honcho_llm_call_inner, where messages is in scope — not the _emit_llm_call_completed helper, which doesn’t receive messages): when TELEMETRY_TRACE_PAYLOADS is on and call_purpose ∈ TELEMETRY_TRACE_PURPOSES (empty = all), build + emit trace.content for each not-yet-emitted-this-run unique message, plus one llm.call.traced. Best-effort, swallow errors (matches the existing :134 pattern).
  • Streaming: emit the llm.call.traced at stream finalization — the wrapper already assembles the full response to return it; capture the assembled output there. On a partial/aborted stream, still emit with finish_reason=cancelled|error, was_truncated=true, and the partial assembled output. (Closes the G2 streaming gap.)
  • Entrypoints (deriver task, dialectic query, dream, summary, agent loop): mint a trace_id (nanoid) if absent and thread it; reuse today’s run_id as root span_id. The summarizer’s placeholder run_id="deriver" and the deriver’s missing run_id are handled here (mint a real root span).
  • step_seq / attempt: threaded explicitly on LLMTelemetryContext, owned by the span entrypoint (§4). No contextvar.
  • Separate emitter instance for the trace stream: a second TelemetryEmitter so a trace burst can never evict billing events from the metrics buffer. Bounded buffer (drop-oldest) with a trace_events_dropped counter so loss is measured, never silent. Durability is guaranteed at the receiver (idempotent upsert), not by an in-process disk spool. Routed to TELEMETRY_TRACE_ENDPOINT (falls back to TELEMETRY_ENDPOINT).

8. Configuration (additive; all default-off / inert)

New fields on TelemetrySettings (env_prefix="TELEMETRY_"):

SettingDefaultPurpose
TRACE_PAYLOADSFalsemaster toggle for payload capture
TRACE_ENDPOINTNoneseparate sink for payloads; falls back to ENDPOINT
TRACE_MAX_BYTES262144per-message cap; truncate + set was_truncated
TRACE_PURPOSES[]allowlist of CallPurpose; empty = all

TRACE_BUFFER_SPOOL_DIR from the prior draft is dropped (disk-spool deferred to OQ2). With TRACE_PAYLOADS=False (default), zero behavior change — purely additive (G5).

9. Local mode: offline importer → Parquet

No live HTTP collector service (NG6). Storage substrate is Parquet; the query engine on top is swappable (DuckDB locally; DuckDB/ClickHouse in cloud via Xatu).

  • Local traces are captured (to a file/dir, or the existing sink). A one-pass honcho-collector import reads the CloudEvents and writes Parquet using the same schema Xatu produces (D2 shared-schema), then DuckDB queries the Parquet directory. No always-on service, no concurrent-writer hazard (Parquet writers don’t contend; each pass writes immutable files, compaction merges later — reusing Xatu’s compaction.py pattern).
  • Parquet layout (one schema, both environments): input_message_refs and tool_schema_refs are list columns (Parquet supports them), so the reconstruction query reads them directly — no body JSON extraction.
-- full timeline of a trace (per-branch order)
SELECT span_id, parent_span_id, step_seq, type, iteration, attempt
FROM events WHERE trace_id = ? AND tenant_id = ?
ORDER BY span_id, step_seq;
 
-- exact context window at iteration N of a span
SELECT c.role, c.content
FROM events e
CROSS JOIN UNNEST(e.input_message_refs) WITH ORDINALITY AS r(hash, ord)
JOIN content c ON c.content_hash = r.hash
WHERE e.span_id = ? AND e.iteration = ? AND e.tenant_id = ?
ORDER BY ord;

(input_message_refs is a real Parquet list column here, so UNNEST works as written.)

10. Relationship to existing pipelines + sunset roadmap

SystemRole todayAfter this spec
CloudEvents metricsanalytics / billing / costunchanged (G5)
REASONING_TRACES_FILE JSONLfull-payload file → minccinounchanged on landing; sunset trigger below
Langfuse (@observe spans)hosted span UI / evalsunchanged on landing; demotion trigger below
llm.call.traced + importer—NEW
reasoning-traces.md Postgres(draft)SUPERSEDED for storage (§11)

Sunset roadmap (the “additive” in G5 is a phase, not forever):

  • Retire JSONL when the CloudEvents path reaches replay-grade parity (§6.4) and minccino consumes the Parquet store instead of traces.jsonl. Trigger: minccino recipe reads Parquet + one eval run reconciles byte-for-byte against JSONL.
  • Demote Langfuse to optional once the local query layer + a thin read view cover the inspection use case. Langfuse stays available but is no longer the default audit surface.

11. Why not Honcho’s Postgres (reasoning-traces.md superseded)

Trace payloads are an append-only, write-heavy, bulk-read OLAP workload; Honcho’s Postgres is OLTP on the synchronous /chat path. Putting hundreds-of-KB context windows there means table bloat, autovacuum/WAL pressure, fatter backups, and connection contention on the one DB that gates chat latency — and it couples a droppable artifact’s failure domain to an un-droppable one. The event stream + Parquet is the right substrate.

reasoning-traces.md’s genuinely distinct value was provenance + agent introspection (Message→Trace→Observation links and a get_reasoning_trace tool). That is deferred to a follow-on that, if a product feature needs it, adds a payload-free reasoning_traces row (ids + token counts + content_hash refs + observation_ids) that points at the event store — metadata in Postgres, bytes in the stream, never both.

12. Phasing (correlation first — corrected order)

  1. Correlation upgrade — mint trace_id at every entrypoint; thread span_id/step_seq/ attempt explicitly on LLMTelemetryContext; keep run_id as span_id. (Must precede emit — payload events need the span-tree fields to exist.)
  2. Events + emit — add llm.call.traced + trace.content; content-addressed (emit-once + content-addressed id); replay-grade output; streaming finalization capture; separate bounded emitter instance + drop metric; behind TRACE_PAYLOADS. Register in events/__init__.py + the closed-taxonomy lint (llm.py:26, tests/telemetry/test_events.py:237).
  3. Local importer — honcho-collector import: CloudEvents → Parquet (Xatu schema) → DuckDB query; docker-compose wiring.

(Forking — parent_span_id population for parallel subagents / dreamer forks — lands when those features ship; §4 already accommodates it.)

13. Follow-on specs (out of scope here)

  • Thin Postgres provenance row + get_reasoning_trace agent tool (§11) — if/when agent self-introspection becomes a product goal.
  • Groudon dashboard exposure — bound by the §6.3 access invariant + Honcho-prompt redaction.
  • Excadrill / eevee reuse — emit llm.call.traced from eval runs; query the Parquet store.
  • Autoresearch loops — agents querying the Parquet trace store to detect regressions and propose experiments.

14. Open questions

  • OQ1: Tool schema capture — content-addressed full schemas (chosen) vs names + schema hash. Spec assumes full schemas content-addressed via tool_schema_refs.
  • OQ2: Disk-spool for the trace emitter — deferred; revisit only if trace_events_dropped fires in practice (on Fly’s ephemeral disk a spool is only partially durable anyway).
  • OQ3: Per-namespace hard deletion over globally content-addressed rows (reference accounting at the Xatu/namespace layer).
  • OQ4: Parquet retention/compaction policy for local mode (size cap, TTL, rollover).

GSTACK REVIEW REPORT

ReviewTriggerWhyRunsStatusFindings
CEO Review/plan-ceo-reviewScope & strategy0——
Codex Review/codex reviewIndependent 2nd opinion1issues_found18 raised, 11 folded, 4 confirmed review findings, 3 corrections
Eng Review/plan-eng-reviewArchitecture & tests (required)1CLEAR11 issues, 0 critical gaps
Design Review/plan-design-reviewUI/UX gaps0——
DX Review/plan-devex-reviewDeveloper experience gaps0——

CODEX: Read the actual telemetry code and found the spec’s headline claim (traced↔metrics join) broken three ways (#3/#4/#5), the “dedupe for free” idempotency misconception (#6), a role-in-hash collision bug (#7), unaddressed streaming (#10), and replay-fidelity gaps (#8/#9). All folded into the rewritten spec.

CROSS-MODEL: Strong agreement, no tension. Codex independently confirmed 4 review findings (step_seq contextvar, disk-spool contradiction, content-table tenant trap, core-only scope creep). Remaining Codex findings were additive (new problems), not disagreements — each was put to the user via AskUserQuestion and folded on approval; none auto-incorporated.

VERDICT: ENG CLEARED — spec rewritten to as-reviewed state, scope reduced to core (emit + correlation + offline importer), all 11 decisions folded. Ready to implement (T1→T7). CEO/Design reviews not required (backend/telemetry, no UI, no product-direction change).

NO UNRESOLVED DECISIONS

Graph View

Backlinks