Plastic Labs Internal Wiki

Recently Updated

cloudevents-enrichment

46 min read

PLAN.md — Full CloudEvents Instrumentation for Honcho

Branch: rajat/add-cloudevents Base: main Goal: Land maximalist telemetry across LLM calls, the deriver, dialectic and dream agents, tool calls, search, and API ingestion — without changing event semantics on existing consumers in a breaking way.

Rev 2 — incorporates review feedback: corrected config names, fixed AgentIteration emission site (was missing no-tool iterations), moved tool-call emission to create_tool_executor (has ToolContext), corrected ENVIRONMENT placement (was Sentry-only), corrected dream scheduler function name (check_and_schedule_dream), corrected summarizer token sources (live in summarize_if_needed, not _create_summary), added outcome field for LLM call failures, added embedding telemetry phase, added emitter-health metrics, request-correlation id, and schema/PII lint tests.

Rev 3 — adds Phase −1 (downstream compatibility spike) so envelope/extension changes are verified against Xatu before producers ship; makes HIGH_VOLUME_SAMPLE_RATE mandatory from day one for llm.call.completed / embedding.call.completed / agent.iteration / agent.tool.call.completed; replaces synthetic outcome="retry_exhausted" with is_final_attempt on the last actual attempt; widens run_id to full UUID/ULID; switches honchotraceid to a pure UUID (path-derived request_id leaks identifiers); sharpens provider/transport semantics; locks call_purpose to a closed taxonomy; renames AgentToolCallEvent → AgentToolCallCompletedEvent; adds result_chars_before_truncation; clarifies dream documents-count timing; adds a §4.5 data-classification table; documents the Xatu migration sequence explicitly.

Rev 4 — Xatu owner confirmed: (a) EnrichedEvent Pydantic model silently drops all CloudEvent extension attrs — honcho* extensions cannot reach storage; (b) tenant attribution is Xatu-side via Fly app name → HonchoInstance.app_name, not Honcho-side; (c) model is analytics-only in Xatu, not a billing key; (d) Xatu archives to S3 Parquet, not ClickHouse. Design pivot: identity fields move from CloudEvent extensions into a BaseEvent.metadata: EventMetadata nested object. Drops honchotenantid entirely. Adds explicit “set TELEMETRY_NAMESPACE to Fly app name at deploy” requirement. Documents Redis-dedupe fail-open behavior. Removes ClickHouse references (was wrong assumption).

Rev 5 — corrects a false premise in rev 4: event.data is JSON-serialized as pa.string() in xatu/consumer/parquet_writer.py:62-75,142, not a typed pa.struct(...). So event.data.metadata.environment is NOT a free nested-column round-trip — it lives as text inside a JSON string and analytics queries need json_extract(data, '$.metadata.environment') (Athena/DuckDB/Trino). Functional but slower than typed columns. The “no Xatu code change required” claim was overstated. Reframes Open Question #5 from “optional follow-up” to “recommended Xatu PR” for ergonomic queries — switching data to pa.struct(...) or extracting metadata.* to top-level columns. Also fixes: run_id form is str(uuid.uuid4())[:8] (with dashes, not pure hex); summarizer functions are _create_and_save_summary (line 363) + _create_summary (line 520) wrapped by summarize_if_needed (line 250); Parquet dedupe is per-batch (same batch overwrites same S3 key) — cross-batch dupes still possible; wires TELEMETRY.ENVIRONMENT to the same source as SentrySettings.ENVIRONMENT.

Rev 6 (post-merge of main into rajat/add-cloudevents) — line-number drift and one substantive code change picked up from main:

  • src/embedding_client.py:68-71 now uses tiktoken.encoding_for_model(self.model) with cl100k_base fallback (was tiktoken.get_encoding("o200k_base") pre-merge). Phase 7’s input_tokens_estimate framing is even more justified — cl100k_base is OpenAI’s older tokenizer (ada-002 era) and is a strict approximation for Gemini and for modern OpenAI models that prefer o200k_base.
  • src/crud/message.py::create_messages now skips empty/whitespace content before calling embedding_client.batch_embed. Phase 7 emit site is unchanged but note that create_messages events without any non-blank content produce zero embedding events.
  • Line-number adjustments in src/llm/tool_loop.py: set_current_iteration is now at line 328 (was 327); the two stream_final_response call sites are at 283 and 412 (were 278 and 408); the max-iteration synthesis call is _final_call at line 439, executed at line 470.
  • Custom-instructions feature (commit a4202641) is not yet merged into main — estimate_minimal_deriver_prompt_tokens() is still the helper at deriver.py:101. Phase 4 needs a forward-compat note: when custom-instructions lands, prompt_scaffold_tokens should source from the wrapped estimate_deriver_prompt_tokens(custom_instructions) instead.

Rev 9 (per spec-owner pass):

  • Schema-version policy refined: one bump per release per changed event. Multiple changes in the same release ride along the single bump; bumps happen even on purely additive changes because the version number is a release-tracking key, not a per-field signal.
  • Per-phase bump intentions clarified. RepresentationCompletedEvent was already bumped v1 → v2 on this branch (for total_input_tokens); Phase 4 additions ride along in v2 — no Phase 4 bump to v3. DreamRunEvent v1 → v2 (Phase 5). DreamSpecialistEvent v1 → v2 (Phase 5). AgentToolSummaryCreatedEvent v1 → v2 (Phase 6). New events (LLMCallCompletedEvent, AgentToolCallCompletedEvent, EmbeddingCallCompletedEvent) start fresh at v1.
  • *Finished → *Completed for naming consistency with the existing event surface. LLMCallCompletedEvent → LLMCallCompletedEvent (llm.call.completed). AgentToolCallCompletedEvent → AgentToolCallCompletedEvent (agent.tool.call.completed, follows the agent.tool.{noun}.{verb} pattern of the four state-change tool events). EmbeddingCallCompletedEvent → EmbeddingCallCompletedEvent (embedding.call.completed).
  • These new *.completed events also emit on failure (via outcome: Literal["success", "error"]), unlike the existing *.completed events which today only fire on success. Convention bend is acknowledged; industry usage (HTTP/function-completion events) accepts this and outcome disambiguates.

Rev 8 (significant simplification per spec-owner review):

  • Envelope barely changes. Drops the proposed BaseEvent.metadata: EventMetadata nested object entirely. Identity reduces to a single field: honcho_version injected by the emitter into the serialized body at emit time. No git_sha (deferred — trivial to add later if a deploy-forensics question comes up), no separate service_version (collapsed into honcho_version), no trace_id (no use case — request_id in track_request is for DB-tracing only, not cross-event correlation), no envelope_version (nothing to version), no honcho* CloudEvent extensions (Xatu drops them; tenant resolution already works via TELEMETRY_NAMESPACE = <fly_app_name>).
  • Phase −1 reduced to “verify, don’t change.” Fly-app-name → tenant mapping is already correctly configured in every Honcho instance per spec owner; dedupe is working. Phase −1 collapses to a confirmation step + volume/$/month hand-off to Xatu owner.
  • RepresentationCompletedEvent: keep input_tokens as-is (queued-message tokens — the billing-resolution key). Don’t rename. The new Phase 4 fields are additive alongside it. Drop the redundant queued_message_tokens field since input_tokens already covers that meaning.
  • Tool-less max_input_tokens truncation promoted from follow-up to a Phase 4 step. Implementing this in src/llm/api.py unlocks a real hit_input_token_cap flag for the deriver. Re-adds the field that rev 7 had deferred.
  • Confirmed event-version baseline against v3.0.6 git tag: every event was at _schema_version=1 in v3.0.6. This branch already bumped representation.completed to v2 (for total_input_tokens). Phase 4 going to v3 is correct math (v1 → v2 → v3). All other planned bumps (DreamRunEvent v1→v2, DreamSpecialistEvent v1→v2, AgentToolSummaryCreatedEvent v1→v2) are correct against the v3.0.6=v1 baseline.
  • Removes EventMetadata from §4.5 classification, §5 emitter-envelope tests, the §4.1 envelope-exempt carve-out, and metadata.* references elsewhere.
  • Open Question 5 (Xatu PR for typed metadata columns) is moot — no metadata object to extract. Removed from open questions.

Rev 7 — fixes stale text and design inconsistencies caught in post-merge review:

  • §2 row 11 still said to emit honcho* CloudEvent extension attrs (contradicts Rev 4). Fixed.
  • §4.1 now explicitly exempts BaseEvent.metadata from per-event schema bumps (envelope-versioned via metadata.envelope_version); without this, Phase 0 would force a v(N+1) bump on every event class.
  • Phase 4 schema block had max_input_tokens + hit_input_token_cap; prose said to omit hit_input_token_cap and rename to configured_max_input_tokens. Schema block now matches prose.
  • Open Question 5 reframed from “optional” to “recommended” — Xatu PR to extract metadata.* to top-level Parquet columns is needed if calibration filters on env/git_sha/trace_id. Stale “nested Parquet columns” wording removed.
  • §4.5 data classification: workspace_name reclassified as user-controlled and potentially sensitive; removed the “doubles as honchotenantid” note (false post-Rev 4).
  • §5 test plan: renamed LLMCallCompletedEvent → LLMCallCompletedEvent, AgentToolCallEvent → AgentToolCallCompletedEvent; dropped the honchotenantid schema lint assertion.
  • Phase 7 stale o200k_base text deleted (Rev 6 covered the model-aware tokenizer).
  • ModelTransport is Literal["anthropic", "openai", "gemini"] — OpenRouter is hidden behind base_url on the openai transport. Phase 1’s provider_label justification updated; transport="openrouter" example removed.
  • Open Question 6 (Persian/Kyogre traceparent) moved out of scope to keep the project boundary tight.
  • New: emitter splits events_dropped_total (data loss) from events_sampled_out_total (intentional sampling) so the former remains a real alert signal.
  • New: emitter populates metadata at serialization time (build dict → inject → emit), not by mutating the event instance — keeps emit(event) side-effect free for tests/callers.
  • run_id widening checklist now includes event schema docs and tests/telemetry/conftest.py fixtures (run_id="abc12345" strings need to be replaced with full ULIDs in fixture data).

1. State of the branch today

1.1 What this branch already adds on top of main

Branch is now merged with main (merge commit fec0f653).

Telemetry-specific commits:

  • f84f4141 feat: add new cloudevents for api routes
  • 40fbd01f fix: add total input tokens to RepresentationCompletedEvent

Net additions:

  • src/telemetry/events/api.py — new module with three events: MessageCreatedEvent, FileUploadedEvent, GetContextEvent. Wired into src/routers/messages.py (POST messages + POST upload) and src/routers/sessions.py / src/routers/peers.py (both context endpoints).
  • RepresentationCompletedEvent (schema v2): gained total_input_tokens (full prompt-side input as seen by the LLM, vs. the existing input_tokens which only counts new-message tokens).
  • Tests: tests/telemetry/test_events.py, tests/telemetry/conftest.py, tests/routes/test_messages.py, tests/integration/test_telemetry.py updated to cover the three new API events.

Relevant changes pulled in from main during merge:

  • 5de8a3b8 fix: use model-aware tokenizer and skip empty messages — embedding_client.py switched from tiktoken.get_encoding("o200k_base") to tiktoken.encoding_for_model(self.model) with cl100k_base fallback. crud/message.py::create_messages now skips blank-content messages before calling batch_embed.
  • 1478cbf1 fix: levels merging in src/config
  • a4ae3729 fix: internal N+1 query in dialectic agent calls

Not in main yet but on flight branches (worth tracking for Phase 4):

  • PR #609 feat: deriver custom instructions (commit a4202641) — will introduce estimate_deriver_prompt_tokens(custom_instructions) wrapper around estimate_minimal_deriver_prompt_tokens(). Phase 4 has a forward-compat note.

1.2 Event surface area as of this branch

ModuleEventCategoryNotes
events/api.pyMessageCreatedEventapiapi / file_upload source
events/api.pyFileUploadedEventapifile-side only, MessageCreatedEvent emitted alongside
events/api.pyGetContextEventapisession + peer scopes
events/representation.pyRepresentationCompletedEventrepresentationv2 — has total_input_tokens
events/agent.pyAgentIterationEventagentdefined but never emitted (see §3.5)
events/agent.pyAgentToolConclusionsCreatedEventagentemitted in agent_tools._handle_create_observations_impl
events/agent.pyAgentToolConclusionsDeletedEventagentemitted in agent_tools._handle_delete_observations
events/agent.pyAgentToolPeerCardUpdatedEventagentemitted in agent_tools._handle_update_peer_card
events/agent.pyAgentToolSummaryCreatedEventagentemitted in utils/summarizer.py
events/dialectic.pyDialecticCompletedEventdialecticemitted in dialectic/core.py::_log_response_metrics
events/dream.pyDreamRunEventdreamemitted in dreamer/orchestrator.py
events/dream.pyDreamSpecialistEventdreamemitted in dreamer/specialists.py
events/reconciliation.pySyncVectorsCompletedEventreconciliation
events/reconciliation.pyCleanupStaleItemsCompletedEventreconciliation
events/deletion.pyDeletionCompletedEventdeletion

1.3 Envelope & shared infra (today)

src/telemetry/emitter.py::TelemetryEmitter.emit:

  • Generates deterministic event id via BaseEvent.generate_id() (sha256 of type:resource_id:timestamp). Good for dedupe (req 11).
  • Source: /honcho/{namespace}/{category} — namespace is settings.TELEMETRY.NAMESPACE, which falls back to top-level settings.NAMESPACE (defaults to "honcho").
  • dataschema: https://honcho.dev/schemas/{event_type}/v{schema_version} — already version-namespaced.

Confirmed downstream constraints (Xatu owner, rev 4):

  1. Xatu silently drops unknown CloudEvent extension attrs. xatu/ingestion/main.py:151-162 builds EnrichedEvent from a fixed Pydantic field list (specversion, id, source, type, time, datacontenttype, dataschema, data, + parsed namespace, + enriched tenant_id). Anything else — traceparent, partitionkey, custom honcho* attrs — never lands in Parquet or Kafka. Implication: any identity/metadata must travel inside event.data, not as envelope extensions.

  2. Tenant is resolved server-side at Xatu, not by Honcho. xatu/shared/crud.py:65 maps Fly app name → HonchoInstance.app_name → tenant_id. The namespace field on inbound events (parsed from source) drives this lookup. Implication: (a) drop the proposed honchotenantid extension entirely; (b) ensure settings.TELEMETRY.NAMESPACE is set to the Fly app name at deploy time so the existing source-based namespace carries enough for Xatu’s enrichment. workspace_name stays a free-form field inside event.data (it already is) and is NOT a tenant key.

  3. Dedupe contract: xatu/ingestion/dedup.py does SET NX on dedup:{tenant_id}:{event_id} with 48h Redis TTL. Tenant-scoped (same id across tenants both pass — fine for us, our resource_id includes workspace_name). Redis-only path: fail-open during Redis outage — dupes leak into Kafka and S3/Parquet. Downstream idempotency partially compensates:

    • Stripe metering has its own idempotency (identifier=event.id in stripe_client.py:238).
    • Parquet writes use a SHA-256 over the batch’s event ids as the S3 filename (parquet_writer.py:99-104), so the same batch always overwrites the same key. But cross-batch duplicates can still produce two distinct files containing the same event id — billing is safe; raw-event analytics over the S3 archive is not.
    • Defense in depth: any analytics query over the Parquet archive should GROUP BY event.id defensively.

  4. model is not a billing key in Xatu. Stripe meters representation.completed on input_tokens and dialectic.completed as count × reasoning_level. model rides in event.data for analytics only. LiteLLM-parity matters for any separate cost-attribution consumer, not for Xatu.

  5. Xatu archives to S3 Parquet (not ClickHouse). xatu/ingestion/parquet_writer.py. ClickHouse references in earlier revisions of this plan were wrong — drop them. Schema “migration” in Xatu means updating PARQUET_SCHEMA (parquet_writer.py:62) and EnrichedEvent.

  6. ⚠️ event.data is stored as pa.string(), not pa.struct(...) — JSON-encoded blob. xatu/consumer/parquet_writer.py:62-75 defines the data column as pa.string(); parquet_writer.py:142 does json.dumps(e.data) before write. Implication: rev 4’s claim that event.data.metadata.environment is a “free nested-column round-trip” was wrong. The metadata round-trips into Parquet, but as text inside a JSON string. Analytics queries need json_extract(data, '$.metadata.environment') (Athena/DuckDB/Trino syntax). This works but is slower than typed columns and uglier in SQL. See Open Question 5 for the recommended Xatu PR to fix this.

2. Requirements → coverage map

Coverage at a glance. Detailed corrections to “Today” / “Action” wording live in §3 phases — when this row and the phase disagree, the phase wins.

#RequirementTodayPhase
1LLM call event (model, provider, transport, max_output, prov in/out tokens, cache r/w, finish_reason)Missing. All fields exist on HonchoLLMCallResponse / AttemptPlan but no event is emitted.New LLMCallCompletedEvent (covers success + error + retry_exhausted via outcome) emitted from src/llm/executor.py::honcho_llm_call_inner (try/finally so failures are captured). One per provider hit. — Phase 1
2Deriver: prompt / message / extra-context / scaffold / provider_input token breakdownPartial — only input_tokens + total_input_tokens today.RepresentationCompletedEvent (stays v2 — already bumped on this branch) gains the breakdown fields, sourced inside process_representation_tasks_batch. — Phase 4
3Deriver: batch caps and whether they firedMissing. Tool-less path in api.py:307-325 ignores max_input_tokens today.Phase 4 implements tool-less truncation in api.py, then emits real batch_max_tokens, max_input_tokens, was_flush_enabled, hit_batch_token_cap, hit_input_token_cap. QueueBatchResult carries cap-hit flags out of queue_manager. — Phase 4
4Observer fanoutMissing.Add observer_count only (skip hash-of-names — leaky and small). — Phase 4
5Emit agent.iteration for dialectic and dreamsDefined but never emitted.Emit directly after each response = await call_func() in execute_tool_loop (covers no-tool terminating iteration AND the final synthesis call after max-iteration). Telemetry context flows from caller via LLMTelemetryContext. — Phase 2
6Tool call events (run_id, iteration, agent_type, tool_name, result_chars, result_tokens_estimate, was_truncated, embedding_query_count)Partial — state-changing tools have dedicated events; read tools (search_*, get_recent_history, etc.) emit nothing.New AgentToolCallCompletedEvent from inner execute_tool closure of create_tool_executor (where ToolContext and error state are already in scope). Includes tool_call_seq so two calls to the same tool in one iteration have unique resource IDs. — Phase 3
7Search tool fields (query_tokens, top_k, results_count, result_tokens, used_embedding)None.Folded into AgentToolCallCompletedEvent as optional fields. Search handlers populate via ToolResult.metadata. — Phase 3
8Summary event: previous_summary_tokens, message_tokens, prompt_scaffold_tokens, provider_input_tokensAgentToolSummaryCreatedEvent v1 only has input_tokens/output_tokens.v2 adds the four new fields (additive — input_tokens kept for back-compat). Sourced in summarize_if_needed (NOT inside _create_summary) and passed down. — Phase 6
9Dream event: dream_type, enabled_types_count, trigger reasons, documents_since_last_dream, document_thresholdMissing.DreamRunEvent v1 → v2 with threshold_reason + delay_reason (split to preserve the two-gate scheduling semantics). Set in check_and_schedule_dream, persisted on the dream queue payload. — Phase 5
10Dream specialist rollups: created/deleted observation counts, peer_card_updated, search_tool_calls_countEmitted today only as separate per-tool events with shared run_id.Denormalize onto DreamSpecialistEvent v1 → v2. Sourced from ToolResult.metadata returned by the create/delete handlers — counting tool names would conflate calls with observations and would include failures/no-ops. — Phase 5 (depends on Phase 3)
11Envelope: tenant_id, workspace_name, environment, service_version, preserve CloudEvent idworkspace_name ✓ in event body. Tenant routing already works server-side at Xatu via Fly app_name (already configured per spec owner). Dedupe on event.id already works.No-op on tenant routing. Per spec-owner pass, environment / service_version were dropped as unused — only honcho_version is added (Phase 0).
12schema_version, honcho_git_sha for calibrationdataschema URL versions per event. Git sha absent.honcho_version injected into event body by emitter. git_sha deferred (3 LOC to add later if needed). Per-event _schema_version already in dataschema URL. — Phase 0

3. Detailed implementation plan

Ordered phases. Each phase is a separate PR-worthy chunk that can ship independently behind no feature flag (events are additive; consumers ignore unknown fields).

Phase −1 — Pre-implementation verification (lightweight)

Why first: before producers start emitting new event types and additional volume, confirm the existing path is healthy and get a $/month estimate so Phase 1’s sample rate can be sized against a real budget.

  1. Confirm TELEMETRY_NAMESPACE matches the Fly app name in every deployed environment. Spec owner has confirmed this is already correct across all instances. Verification only — no change expected.
  2. Confirm TELEMETRY_ENDPOINT is correct per environment. Verification only.
  3. Volume / cost data hand-off to Xatu owner. Xatu owner asked for current EPS by event type so they can project the $/month delta of the new events. Point them at wherever this data lives (Logflare / BigQuery / PostHog / a sheet). Loop back with the estimate before merging Phase 1 so HIGH_VOLUME_SAMPLE_RATE is sized against a real budget.

Files touched: none in src/. Information gathering only.

Risk: zero — pure verification + data hand-off.

Phase 0 — Minimal envelope addition: honcho_version (covers req 12, narrowly)

Why: the only piece of envelope identity the spec owner judged worth carrying is the Honcho version. Tenant routing already works (via TELEMETRY_NAMESPACE = <fly_app_name>). Dedupe already works (Xatu Redis SET NX on event_id). Environment / git_sha / trace_id were excised as unused. So Phase 0 collapses to one field.

  1. TelemetrySettings gains one field:

    HONCHO_VERSION: str | None = None   # TELEMETRY_HONCHO_VERSION (CI/CD); falls back to importlib.metadata.version("honcho")

    Resolution order at first access:

    • TELEMETRY_HONCHO_VERSION env var (CI sets this from the build tag), else
    • importlib.metadata.version("honcho"), else
    • None (emitter omits the field from the body).

  2. Emitter injects honcho_version at serialization time. The event class doesn’t carry the field. This keeps BaseEvent unchanged (no per-event _schema_version bump needed) and keeps event-instance mutation out of emit(event):

    def emit(event: BaseEvent) -> None:
    body = event.model_dump(mode="json")
    version = settings.TELEMETRY.HONCHO_VERSION
    if version:
        body["honcho_version"] = version
    cloud_event = CloudEvent(attributes, body)
    ...

    Test fixtures and integration tests asserting on emitted events stay deterministic — they observe the event instance, not the post-emit body.

  3. What’s NOT in Phase 0:

    • No honcho* CloudEvent extension attrs. Xatu drops them (xatu/ingestion/main.py:151-162 builds EnrichedEvent from a fixed Pydantic field list).
    • No git_sha. Deferred — 3 LOC to add later if a deploy-forensics question arises (env var + body field).
    • No environment. Deferred — could pull from SentrySettings.ENVIRONMENT if needed; not requested.
    • No trace_id. The existing request_id in track_request (src/main.py:235) is a DB-trace log key, not a cross-event correlation id. No use case for emitting it as event metadata.
    • No envelope_version. Nothing to version when the envelope doesn’t change.
    • No honchotenantid. Tenant attribution is already correct server-side at Xatu via Fly app name → HonchoInstance.app_name lookup. Verified by spec owner.
    • No per-event _schema_version bump — honcho_version is emitter-injected into the body, not declared on any event class.

  4. Emitter-health metrics (companion to envelope work, low LOC):

    • Add Prometheus counters in src/telemetry/prometheus/metrics.py:
      • honcho_telemetry_events_emitted_total{type} — events that made it onto the wire.
      • honcho_telemetry_events_sampled_out_total{type} — events deliberately dropped by HIGH_VOLUME_SAMPLE_RATE. Intentional, not data loss.
      • honcho_telemetry_events_dropped_total{reason="buffer_full"|"send_failed"} — unintentional data loss. This is the page-able metric.
      • honcho_telemetry_buffer_size — gauge.
    • Splitting sampled_out from dropped matters operationally: a dropped_total alert should never fire under normal load. If it goes off, it means real telemetry was lost, not just sampled.
    • Increment from TelemetryEmitter.emit (when _buffer is full and deque(maxlen=) evicts → dropped), at sampler entry (when sample-out → sampled_out), and _send_batch (final retry failure → dropped).

  5. Document downstream dedupe contract in src/telemetry/events/__init__.py docstring: consumers must treat CloudEvent id as the primary key.

Files touched: src/config.py, src/main.py (middleware), src/telemetry/emitter.py, src/telemetry/events/__init__.py, src/telemetry/prometheus/metrics.py, src/utils/types.py, new src/_version.py.

Risk: low. Pure additive on the envelope. Existing consumers ignore unknown extensions.

Phase 1 — Per-LLM-call event (req 1)

Why: every other “agent” event is an aggregate; without a per-call event we can’t do per-provider cost calibration or finish_reason analysis.

Naming: LLMCallCompletedEvent with event_type = "llm.call.completed". Consistent with the rest of the event surface; the outcome: Literal["success", "error"] field distinguishes success from failure.

  1. New event src/telemetry/events/llm.py::LLMCallCompletedEvent:

    _event_type = "llm.call.completed"
_schema_version = 1
_category = "llm"
 
workspace_name: str | None         # not always available (system calls)
call_purpose: CallPurpose | None   # closed enum — see (2) below; NOT from track_name
parent_category: str | None        # "representation" | "dialectic" | "dream" | "summary" — for cheaper analytics joins
transport: ModelTransport          # SDK transport: "anthropic" | "openai" | "gemini" — what we're actually talking to (see config.py:25)
provider_label: str | None         # Vendor inferred from base_url + model when a relay is in play (e.g., transport="openai" + base_url pointing at OpenRouter + model="anthropic/claude-..."); None when not reliably inferable
model: str
effective_max_output_tokens: int
provider_input_tokens: int
provider_output_tokens: int
cache_read_tokens: int
cache_creation_tokens: int
finish_reason: str | None
outcome: Literal["success", "error"]
is_final_attempt: bool             # True on the last attempt regardless of outcome — replaces synthetic "retry_exhausted"
error_class: str | None = None     # exception class name when outcome == "error"
attempt: int                       # tenacity attempt index, 1-indexed
retry_attempts: int                # total attempts allowed
was_fallback: bool                 # plan is runtime_model_config.fallback
duration_ms: float
has_tools: bool
tool_call_count: int
was_stream: bool = False           # True for stream_final_response path; token totals partial — see (5)
# Agent correlation
run_id: str | None = None          # full UUID/ULID — see §4.7 on run_id width
iteration: int | None = None       # passed explicitly via LLMTelemetryContext — see (4)

  2. call_purpose is a closed taxonomy, not a free-form string.

    class CallPurpose(str, Enum):
    DERIVER_REPRESENTATION = "deriver.representation"
    DIALECTIC_ANSWER = "dialectic.answer"
    DREAM_DEDUCTION = "dream.deduction"
    DREAM_INDUCTION = "dream.induction"
    SUMMARY_SHORT = "summary.short"
    SUMMARY_LONG = "summary.long"

    track_name is observability text (free-form, drifts); analytics columns need stable slugs. New call sites must add an enum value before they can emit. Schema lint enforces it.

  3. Synthetic retry_exhausted is gone. Instead, the LAST attempt (whether it succeeds or errors) carries is_final_attempt=True. Calibration query for “exhausted” becomes WHERE outcome='error' AND is_final_attempt=true. This removes the awkward case where a synthetic outer event had to invent model/provider/duration_ms it never had.

  4. provider vs transport was redundant. ModelTransport = Literal["anthropic", "openai", "gemini"] (config.py:25) is the SDK we’re talking to. OpenRouter and other relays don’t get their own transport — they’re hidden behind base_url on the openai transport. A separate provider field would duplicate transport most of the time and lie when relays route to a different vendor. Use transport as the authoritative field. provider_label is best-effort vendor inference: parse the model name prefix ("anthropic/claude-..." → "anthropic") or the base_url host (openrouter.ai → relay). When inference fails, leave it None.

    model is analytics-only at Xatu, not a billing key (Xatu bills representation.completed.input_tokens and dialectic.completed count × reasoning_level — see §1.3 point 4). For cost attribution by model in the separate analytics system, model strings must match whatever LiteLLM bills against — that’s an emitter-side contract, not Xatu’s concern. Action: when wiring LLMCallCompletedEvent.model, ensure it’s the same string LiteLLM logs in its usage payload. Spot-check by joining Honcho events to LiteLLM logs on model for a sample window after Phase 1 ships.

  5. Emit site: wrap honcho_llm_call_inner in src/llm/executor.py with a try/finally so we emit on success AND on exception (with outcome="error"). That’s the one-call-one-provider boundary. is_final_attempt is computed from the outer tenacity state — pass it down via LLMTelemetryContext or compute from current_attempt == retry_attempts at emit time. No synthetic outer event needed.

  6. Threading context — honcho_llm_call_inner lacks workspace_name, call_purpose, and primary-vs-fallback identity. Two changes:

    • Add an explicit LLMTelemetryContext dataclass (run_id, parent_category, agent_type, workspace_name, call_purpose, optional peer fields, iteration: int | None). Thread it kwarg-style through honcho_llm_call → honcho_llm_call_inner. The tool loop updates iteration on the context object (or constructs a fresh one) before each inner call.
    • Promote was_fallback and attempt onto AttemptPlan in src/llm/runtime.py — today the executor sees plan.selected_config but has to compare against runtime_model_config.fallback to know if this is the fallback. Putting is_fallback: bool on AttemptPlan and propagating attempt/retry_attempts makes emission a straight read.

  7. Don’t read iteration from set_current_iteration ContextVar. That ContextVar is set at tool_loop.py:328 after the LLM call and after tool execution starts. Reading it from the executor would yield stale values on every LLM call. Instead, the tool loop explicitly sets LLMTelemetryContext.iteration = current_iteration_number before each honcho_llm_call_inner invocation. For non-tool calls (deriver, summarizer), iteration=None.

  8. Streaming gaps (known, called out):

    • stream_final_response at tool_loop.py:283 and :412 makes a fresh LLM call to stream the synthesized answer. Token totals are not knowable until the stream drains.
    • For agentic streams the per-iteration tool-loop emissions cover the pre-stream phase. The final streamed call is missing from LLMCallCompletedEvent accounting.
    • Mitigation in v1: emit a LLMCallCompletedEvent with was_stream=True at stream-setup time with outcome="success" and token fields set to 0. Wire token totals into StreamingResponseWithMetadata and emit a follow-up event on __aiter__ completion as a follow-up. Acceptable until streamed dialectic becomes the cost driver.

  9. Sampler is mandatory from day one — see §4.2 for the cardinality math. LLMCallCompletedEvent participates in HIGH_VOLUME_SAMPLE_RATE. The aggregate envelopes (DialecticCompletedEvent, DreamRunEvent, RepresentationCompletedEvent) are NEVER sampled — those are the calibration ground truth.

  10. Skip emission when response_model is None AND streaming AND no tools (the rare “stream once, return chunks” non-agentic path). Document it.

  11. Budgeting concern: high cardinality (1 event per LLM call → potentially thousands per deriver batch under high load). Mitigation: existing emitter already batches at FLUSH_THRESHOLD=50. Verify under load. If volume becomes a problem, add settings.TELEMETRY.SAMPLE_LLM_CALLS: float = 1.0 knob in a follow-up.

Files touched: new src/telemetry/events/llm.py, src/llm/api.py, src/llm/executor.py, src/utils/types.py (telemetry context vars), src/telemetry/events/__init__.py.

Phase 2 — Agent iteration emission (req 5)

Why: This is defined and unwired, biggest “easy win” gap.

Correction: the existing iteration_callback block in tool_loop.py:367 fires after tool execution, which means the final no-tool iteration (the one that returns at line 256) is skipped. Emit AgentIterationEvent immediately after each LLM response (right after response = await call_func() at line 249), before the no-tool early return. The caller-provided iteration_callback can stay where it is — that’s a different concern (it signals end-of-iteration including tool execution).

  1. New emission sites in src/llm/tool_loop.py:

    • Per-iteration: after line 254 (total_cache_read_tokens += response.cache_read_input_tokens), if LLMTelemetryContext carries agent metadata (run_id + parent_category + agent_type), emit AgentIterationEvent with tool_calls=[tc["name"] for tc in response.tool_calls_made] (empty list if it’s the no-tool terminating iteration). Guarantees N iterations → N events.
    • Max-iteration synthesis call: the loop falls through at tool_loop.py:391+ when it hits max_tool_iterations, then makes a final non-tool LLM call (_final_call defined at line 439, executed at line 470) whose tokens ARE added to running totals (final_response.input_tokens = total_input_tokens + ...). That call has no per-iteration event under the per-while emission. Emit one more AgentIterationEvent after final_response = await final_call_func() with iteration = iteration + 1, tool_calls=[], and the synthesis call’s deltas.
    • These two sites together mean: agent runs that stop naturally get N events; runs that exhaust the loop get N+1 (one extra for synthesis). The iteration field on LLMCallCompletedEvent matches.

  2. Update call sites to set the telemetry context:

    • dreamer/specialists.py::run: pass LLMTelemetryContext(run_id=run_id, parent_category="dream", agent_type=self.name, workspace_name=..., observer=..., observed=...).
    • dialectic/core.py::answer and answer_stream: pass LLMTelemetryContext(run_id=self._run_id, parent_category="dialectic", agent_type="dialectic", workspace_name=self.workspace_name, peer_name=self.observed). Dialectic currently passes no iteration_callback — that’s fine; emission no longer depends on the callback.

  3. Schema: AgentIterationEvent is already correctly shaped. Keep at v1.

Files touched: src/llm/api.py, src/llm/tool_loop.py, src/dialectic/core.py, src/dreamer/specialists.py, src/utils/types.py (telemetry ContextVar).

Phase 3 — Tool call events (req 6, 7)

  1. New event src/telemetry/events/agent.py::AgentToolCallCompletedEvent (single event, search fields optional):

    _event_type = "agent.tool.call.completed"     # symmetric with llm.call.completed / embedding.call.completed
_schema_version = 1
_category = "agent"
 
run_id: str
iteration: int
tool_call_seq: int                  # 0-indexed position within the iteration's tool batch
provider_tool_call_id: str | None   # tool id from the model when available; for cross-event correlation
parent_category: str
agent_type: str
workspace_name: str
tool_name: str
duration_ms: float
result_chars: int                            # bytes of result as returned to the LLM (after any truncation)
result_chars_before_truncation: int | None   # original size when was_truncated=true; None otherwise
result_tokens_estimate: int                  # tiktoken-based size proxy for the result string; estimate only
was_truncated: bool
is_error: bool
 
# Search-specific (None for non-search tools)
query_tokens: int | None = None
top_k: int | None = None
results_count: int | None = None
used_embedding: bool | None = None
embedding_query_count: int = 0
    • get_resource_id() → f"{run_id}:{iteration}:{tool_call_seq}". Without tool_call_seq, the model can legitimately call search_memory twice in one iteration (it does) and we’d collide.
    • Dropped result_tokens (search-specific) — it overlapped with the generic result_tokens_estimate. Search results ARE the tool result, so the generic field covers it.
    • Added result_chars_before_truncation — was_truncated=true alone tells calibration that truncation happened but not by how much. The delta is the signal.
    • Participates in HIGH_VOLUME_SAMPLE_RATE.

  2. Emit site: the inner execute_tool closure inside create_tool_executor at src/utils/agent_tools.py:2102. This is the right place because:

    • ToolContext is already in scope (workspace, run_id, agent_type, parent_category, observer, observed) — no plumbing needed.
    • It already wraps every handler in a try/except that catches Exception, so is_error is trivially known here. tool_loop.py only sees an opaque string back and can’t distinguish errors reliably.
    • Search-specific fields (top_k, results_count, etc.) can be returned by handlers via a richer return type without changing the public tool_executor callable signature.

  3. tool_call_seq + provider_tool_call_id plumbing: create_tool_executor doesn’t natively know the call ordinal within an iteration. The tool_loop does (it iterates for tool_call in response.tool_calls_made:). Two options:

    • (a) Pass at call time: extend the tool_executor callable signature from (tool_name, tool_input) to (tool_name, tool_input, *, seq: int, provider_id: str | None). Cleanest but touches the public contract.
    • (b) ContextVar: mirror set_current_iteration with set_current_tool_call_seq(seq, provider_id). The tool loop sets it before invoking tool_executor, the executor reads it. Less intrusive.
    • Recommend (b) — symmetric with the existing iteration ContextVar pattern and avoids changing tool_executor’s signature.

  4. Handler return type:

    • Today handlers return str. Introduce an internal ToolResult(content: str, metadata: dict | None = None) dataclass returned by handlers.
    • In execute_tool, unwrap before returning the string to tool_loop. Emit AgentToolCallCompletedEvent with metadata fields populated (search handlers fill top_k/results_count/etc.; create/delete handlers fill created_count/deleted_count so Phase 5 can read those for rollups; others leave metadata None).
    • This is a private contract — the public Callable[[str, dict], Any] returned from create_tool_executor is unchanged.

  5. Don’t break the state-changing tool events. The four existing AgentTool* events (created/deleted/peer_card/summary) carry semantic meaning beyond “tool was called.” Keep them. AgentToolCallCompletedEvent is the lightweight per-invocation companion that also covers read-only tools.

Files touched: src/telemetry/events/agent.py, src/utils/agent_tools.py (executor wrapper + per-handler metadata returns for search tools).

Phase 4 — RepresentationCompleted (additive, stays at v2) + tool-less truncation

  1. Keep RepresentationCompletedEvent._schema_version = 2. Per Rev 9 schema policy, additive fields don’t bump. Add fields without renaming the existing input_tokens:

    # Existing — KEEP UNCHANGED:
input_tokens: int            # Queued-message token count. This is the BILLING-RESOLUTION key
                             # in Xatu; do not rename or repurpose. Equivalent to the new
                             # queued_message_tokens conceptually, but kept as-is for downstream
                             # column compatibility.
total_input_tokens: int      # Actual provider-side input_tokens for the LLM call.
output_tokens: int
 
# Req 2: token breakdown (additive — does NOT shadow input_tokens)
queued_message_count: int            # number of queue items processed
prompt_message_count: int            # total messages in the prompt (queued + extra context)
prompt_message_tokens: int           # sum of message token_counts in the prompt
extra_context_message_count: int     # prompt - queued
extra_context_tokens: int            # tokens in the extra-context (interleaved) messages
prompt_scaffold_tokens: int          # estimated tokens in the system/scaffold prompt
# Req 3: caps (real hit flags — see (3) below; tool-less truncation lands in this phase)
batch_max_tokens: int                # REPRESENTATION_BATCH_MAX_TOKENS
max_input_tokens: int                # DERIVER.MAX_INPUT_TOKENS (now actually enforced)
was_flush_enabled: bool
hit_batch_token_cap: bool            # queue batcher truncated to fit batch_max_tokens
hit_input_token_cap: bool            # LLM call truncated to fit max_input_tokens (see (3))
# Req 4: fanout
observer_count: int

    Note: queued_message_tokens is omitted because the existing input_tokens field already means exactly that. Keeping two fields with the same semantic would be a footgun for downstream queries.

  2. Source mapping inside src/deriver/deriver.py::process_representation_tasks_batch:

    • input_tokens (existing) — already computed as messages_tokens at deriver.py:103. Keep as-is.
    • queued_message_count = len(queue_item_message_ids).
    • prompt_message_count = len(messages); prompt_message_tokens = sum(msg.token_count for msg in messages).
    • extra_context_message_count = prompt_message_count - queued_message_count; extra_context_tokens = prompt_message_tokens - input_tokens.
    • prompt_scaffold_tokens = estimate_minimal_deriver_prompt_tokens() (already computed as prompt_tokens at deriver.py:101).
      • Forward-compat: PR #609 (feat: deriver custom instructions, commit a4202641) is in flight on an unmerged branch. When it lands, the helper becomes estimate_deriver_prompt_tokens(custom_instructions) and falls through to estimate_minimal_deriver_prompt_tokens() only when custom_instructions is None. Source prompt_scaffold_tokens from the wrapped helper.
    • batch_max_tokens = settings.DERIVER.REPRESENTATION_BATCH_MAX_TOKENS (src/config.py:742).
    • max_input_tokens = settings.DERIVER.MAX_INPUT_TOKENS.
    • was_flush_enabled: whether deriver was operating in flush-on-cap mode for this batch.
    • hit_batch_token_cap: whether the queue batcher truncated to fit batch_max_tokens.
    • hit_input_token_cap: whether the LLM call truncated input to fit max_input_tokens — see (3) for how this becomes measurable.
    • observer_count = len(observers).

  3. Tool-less max_input_tokens enforcement (NEW — formerly a follow-up).

    • Today the deriver calls honcho_llm_call(..., max_input_tokens=settings.DERIVER.MAX_INPUT_TOKENS) (deriver.py:139) but the tool-less LLM path in api.py:307-325 drops the parameter — only execute_tool_loop consumes it. So the deriver’s cap is configured but never enforced or measurable.
    • Fix in this phase: in src/llm/api.py, when tools / tool_executor is None and max_input_tokens is not None, run messages (or [{"role": "user", "content": prompt}]) through the existing truncate_messages_to_fit(messages, max_input_tokens) helper from src/llm/conversation.py before the inner call. Return both the (possibly truncated) message list AND a was_truncated boolean to the caller — easiest path is to add a kwarg report_truncation: bool = False that, when set, returns HonchoLLMCallResponse augmented with a input_was_truncated: bool field.
    • Deriver consumes the flag: after response = await honcho_llm_call(...), set hit_input_token_cap = response.input_was_truncated.
    • This is a ~50 LOC change in api.py and a new field on HonchoLLMCallResponse. Worth it: it converts the deriver’s advisory cap into a real, enforced, observable cap.

  4. Plumbing for hit_batch_token_cap and was_flush_enabled: rather than inferring from inputs after the fact, have queue_manager.get_queue_item_batch return a QueueBatchResult dataclass with the batch + cap-hit flags.

  5. Update fixtures in tests/telemetry/conftest.py::sample_representation_event and any integration tests asserting payload shape.

Files touched: src/telemetry/events/representation.py, src/deriver/deriver.py, src/deriver/consumer.py (pass QueueBatchResult through), src/deriver/queue_manager.py (return QueueBatchResult), src/llm/api.py + src/llm/types.py (tool-less truncation + input_was_truncated on response). Tests.

Risk: medium — QueueBatchResult is a moderate refactor; tool-less truncation touches the LLM API surface but uses existing helpers.

Phase 5 — Dream events (req 9, 10)

  1. DreamRunEvent v1 → v2. Adds:

    dream_type: str                              # "omni" today; future: "deductive" | "inductive"
enabled_types_count: int                      # len(settings.DREAM.ENABLED_TYPES)
threshold_reason: str                         # "document_threshold" | "manual" | "surprisal"
delay_reason: str                             # "idle_timeout" | "immediate" | "min_hours_gate"
documents_since_last_dream_at_schedule: int   # value at the moment check_and_schedule_dream made the decision
document_threshold: int                       # settings.DREAM.DOCUMENT_THRESHOLD at schedule time

    Time-disambiguated naming: the count of “documents since last dream” changes between when the dream is scheduled and when it actually fires (idle delay). documents_since_last_dream alone is ambiguous. Naming it _at_schedule makes it clear it’s a snapshot, not a live count.

    Correction: the actual scheduler entrypoint is check_and_schedule_dream (src/dreamer/dream_scheduler.py:219), not should_dream_for_collection. And scheduling is two gates: (a) threshold reached AND MIN_HOURS_BETWEEN_DREAMS passed, then (b) IDLE_TIMEOUT_MINUTES delay before the dream actually fires. A single trigger_reason field flattens that distinction. Split into threshold_reason (what triggered the schedule) and delay_reason (what governed when it fired) — preserves both gates for calibration.

    Plumbing: set both at schedule time in check_and_schedule_dream, persist on the dream queue payload (src/utils/queue_payload.py), thread through consumer.process_dream → orchestrator.run.

  2. DreamSpecialistEvent v1 → v2. Adds denormalized rollups:

    created_observation_count: int    # actual observations created across all create calls
deleted_observation_count: int    # actual observations deleted across all delete calls
peer_card_updated: bool           # at least one successful update_peer_card call
search_tool_calls_count: int      # call count (this one IS just counting names — search tools always run something)

    Correction: counting tool-call names gives call count, not observation count. A single create_observations call can create N observations or zero (if all fail validation). Use the ToolResult.metadata produced by handlers in Phase 3 — create_observations handler returns metadata["created_count"], delete_observations returns metadata["deleted_count"], etc.

    Plumbing:

    • The handler’s metadata flows out through ToolResult into AgentToolCallCompletedEvent. To make specialist rollups not require querying the event store, also retain the metadata on response.tool_calls_made[i] — extend tool_loop’s all_tool_calls.append(...) to include tool_result_metadata. Then in specialists.py::run: 
      created = sum(tc.get("tool_result_metadata", {}).get("created_count", 0)
              for tc in response.tool_calls_made
              if tc["tool_name"] == "create_observations")
deleted = sum(...)
peer_card_updated = any(tc["tool_name"] == "update_peer_card"
                        and tc.get("tool_result_metadata", {}).get("success")
                        for tc in response.tool_calls_made)
search_tool_calls_count = sum(1 for tc in response.tool_calls_made
                              if tc["tool_name"] in {"search_memory","search_messages","search_messages_temporal"})
    • search_tool_calls_count stays as a name-count because search tools always execute (the metadata-based count is the same as the name-based count).

Files touched: src/telemetry/events/dream.py, src/dreamer/orchestrator.py, src/dreamer/specialists.py, src/dreamer/dream_scheduler.py, src/utils/queue_payload.py, src/deriver/consumer.py.

Phase 6 — Summary event v2 (req 8)

  1. AgentToolSummaryCreatedEvent → v2 fields, additive:

    # Existing (KEEP for backward compat):
input_tokens: int   # message-derived token estimate (what's already there)
output_tokens: int
# New:
previous_summary_tokens: int
message_tokens: int                # = current input_tokens semantic, but explicit
prompt_scaffold_tokens: int
provider_input_tokens: int         # actual provider input_tokens from HonchoLLMCallResponse

    Correction: keep input_tokens for backward compatibility (don’t repurpose its meaning) and add provider_input_tokens alongside. Consumers can migrate at their own pace; schema_version=2 signals the new fields are available.

  2. Source location: there are three functions, not two — the spec earlier conflated them:

    • summarize_if_needed (summarizer.py:250) — the entry point; computes input tokens, decides whether short/long summary is needed.
    • _create_and_save_summary (summarizer.py:363) — the wrapper that holds messages_tokens, calls estimate_short_summary_prompt_tokens() / estimate_long_summary_prompt_tokens(), and where the existing emit(AgentToolSummaryCreatedEvent(...)) lives.
    • _create_summary (summarizer.py:520) — the innermost LLM call; returns (Summary, is_fallback, llm_input_tokens, llm_output_tokens).

    The new fields (previous_summary_tokens, message_tokens, prompt_scaffold_tokens, provider_input_tokens) all need to be in scope where emit is called — i.e. inside _create_and_save_summary. Plumb them there:

    • messages_tokens → already a local in _create_and_save_summary.
    • previous_summary_tokens → tiktoken count of previous_summary_text at the same scope; compute alongside the existing prompt_tokens calculation.
    • prompt_scaffold_tokens → estimate_short_summary_prompt_tokens() / estimate_long_summary_prompt_tokens() (already called nearby).
    • provider_input_tokens → llm_input_tokens returned from _create_summary.

    No need to move the emit site — just enrich the existing call.

Files touched: src/telemetry/events/agent.py, src/utils/summarizer.py, tests.

Phase 7 — Embedding telemetry (review extension)

Why: embedding calls are real provider spend ($-per-token like the LLM) but invisible today. Search tools, observation creation, message-embedding sync, agent tools, and dreamer all hit the embedding API.

  1. New event src/telemetry/events/llm.py::EmbeddingCallCompletedEvent:

    _event_type = "embedding.call.completed"
_schema_version = 1
_category = "llm"
 
workspace_name: str | None
call_purpose: EmbeddingCallPurpose | None  # closed enum: "search_memory" | "search_messages" | "create_observations" | "vector_sync" | "summary" | None
parent_category: str | None                # "representation" | "dialectic" | "dream" | "reconciliation" — for joins
provider: str                              # "openai" | "gemini"
model: str
input_count: int                           # number of texts embedded in this call
input_tokens_estimate: int                 # tiktoken via encoding_for_model(model) with cl100k_base fallback; ESTIMATE only
batch_size: int                            # how many texts batched into one provider call
duration_ms: float
outcome: Literal["success", "error"]
is_final_attempt: bool                     # mirrors LLMCallCompletedEvent contract
error_class: str | None = None
run_id: str | None = None                  # full UUID/ULID when called from an agentic run

    • Closed enum for call_purpose (symmetric with LLMCallCompletedEvent taxonomy decision).

    • parent_category lets analytics join embedding.call.completed ↔ llm.call.completed ↔ agent.iteration ↔ agent.tool.call.completed without inferring the run shape from event type alone.

    • Participates in HIGH_VOLUME_SAMPLE_RATE.

    • input_tokens_estimate semantics (post-merge): embedding_client.py:68-71 uses tiktoken.encoding_for_model(self.model) with a cl100k_base fallback. That’s correct for older OpenAI embedding models (ada-002 era) but a strict estimate for: (a) newer OpenAI embedding models that prefer o200k_base, (b) Gemini, which has its own tokenizer entirely. Treat the field as a cost-attribution approximation, not a billing-exact number. If billing precision matters downstream, prefer the provider’s own returned token count when available (OpenAI returns usage.total_tokens on embedding responses; Gemini does not).

    Naming: field is input_tokens_estimate, not input_tokens — see the input_tokens_estimate semantics note above for why this is an estimate.

  2. Emit site: wrap each provider call in src/embedding_client.py. The client already has provider, model, batching logic, and the tiktoken estimator.

  3. call_purpose via ContextVar, not manual threading. There are many embedding_client call sites well beyond agent_tools.py (CRUD paths, dreamer, scripts, vector sync). Threading call_purpose through every caller is brittle:

    • Add embedding_call_purpose ContextVar in src/utils/types.py.
    • Provide a set_embedding_call_purpose(name) helper + with embedding_call_purpose("search_memory"): context manager.
    • Search/observation/sync call sites set it locally; the embedding client reads it. Missing = None, which is fine.
    • This is the same pattern as the planned LLMTelemetryContext in Phase 1 — symmetric.

  4. Risk: medium-to-high cardinality. Phase 1 said LLM call events were the only volume risk; embedding events are comparable. Vector sync batches into large requests but interactive paths (search_memory / search_messages) emit one event per query. With search-heavy dialectic this matches or exceeds the LLM call rate.

    • Update §4.2 to reflect this.
    • The sampler knob (originally proposed as LLM_CALL_SAMPLE_RATE) becomes a unified settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE: float = 1.0 covering both llm.call.completed and embedding.call.completed. Sampler is deterministic per run_id when available so traces remain coherent.

Files touched: new event class, src/embedding_client.py, src/utils/types.py (ContextVar + context manager), src/utils/agent_tools.py (search handlers wrap embedding calls in the context manager), src/crud/representation.py, vector-sync code path.

4. Cross-cutting concerns

4.1 Schema evolution rules

  • Document in src/telemetry/events/__init__.py and reference here.
  • Additive fields only. Make them required only when there’s a clean default; otherwise default= something benign (0, False, None).
  • Bump _schema_version once per release for any event that changed in that release. Multiple additive changes that ship together don’t compound the bump — they’re all part of the same vN+1 release. Bump happens regardless of whether the change is additive or breaking; the version number is a release-tracking key, not a per-field signal.
  • Never rename or remove a field without a v(N+1) and explicit downstream coordination.
  • Emitter-injected body fields (currently just honcho_version — see Phase 0) are NOT declared on any event class and therefore do not trigger schema bumps. Consumers must tolerate the additional top-level body key.
  • Practical consequence for this plan (baseline = v3.0.6 = everything at v1):
    • RepresentationCompletedEvent is already at v2 on this branch (total_input_tokens was added). Phase 4’s additional fields ride along in v2 — same release, no double-bump.
    • DreamRunEvent v1 → v2 (Phase 5 additions).
    • DreamSpecialistEvent v1 → v2 (Phase 5 rollups).
    • AgentToolSummaryCreatedEvent v1 → v2 (Phase 6 token breakdown).
    • New events introduced in Phases 1, 3, 7 (LLMCallCompletedEvent, AgentToolCallCompletedEvent, EmbeddingCallCompletedEvent) start fresh at v1.

4.2 Cardinality / volume

Sampler ships day-one. Phase 1, Phase 3, and Phase 7 cannot be merged unsampled — the math doesn’t work.

Per-dialectic-request rough shape with the new events:

  • ~5 LLM iterations (llm.call.completed × 5)
  • ~5 iteration markers (agent.iteration × 5)
  • ~5 tool calls across those iterations (agent.tool.call.completed × 5)
  • ~3 embedding calls (embedding.call.completed × 3)
  • 1 aggregate (dialectic.completed × 1)

→ ~19 events per dialectic request. At 1k dialectic qps that’s ~19k events/sec. Current emitter MAX_BUFFER_SIZE=10000 fills in ~500ms. Drop rate is non-zero even at moderate load.

Hard rules:

  • settings.TELEMETRY.HIGH_VOLUME_SAMPLE_RATE: float = 0.1 is the default, applied to llm.call.completed, embedding.call.completed, agent.iteration, agent.tool.call.completed. Tune per environment.
  • Sampling is deterministic on run_id — sample = hash(run_id) % 100 < rate * 100. Either all events for a run are sampled or none, so a sampled run is fully reconstructible end-to-end. Calls without run_id (e.g. summarizer, vector sync) sample independently per call.
  • Aggregate envelopes (RepresentationCompletedEvent, DialecticCompletedEvent, DreamRunEvent, DreamSpecialistEvent, MessageCreatedEvent, FileUploadedEvent, GetContextEvent) are always emitted at rate=1.0. They carry the rolled-up totals — calibration ground truth never relies on the sampler.
  • Schema lint enforces this: every high-volume event class carries a _volume_class = "high" ClassVar; every aggregate carries _volume_class = "ground_truth". The sampler reads _volume_class.

Monitoring:

  • Watch honcho_telemetry_events_dropped_total{reason="buffer_full"} (Phase 0). Any non-zero is silent data loss in raw events.
  • Watch honcho_telemetry_buffer_size gauge against MAX_BUFFER_SIZE. Alert at >80% sustained.
  • A synthetic load benchmark (tests/telemetry/test_load.py) verifies the emitter survives 20k events/sec sustained.

4.3 Error budget

  • Telemetry emission is already wrapped in try/except + sentry_sdk.capture_exception in events/__init__.py::emit. Stay within that pattern — no instrumentation should be able to raise into the request path.
  • For envelope changes in phase 0: if GIT_SHA resolution fails at startup, log + continue with unknown — don’t fail boot.

4.4 Backward compatibility for consumers — Xatu rollout sequence

Xatu stores events as Parquet in S3 (xatu/ingestion/parquet_writer.py). The data column is a JSON-serialized string (pa.string()), so all event-payload fields — including new ones we add — round-trip cleanly without Parquet schema changes, but queries read them via json_extract(data, '$.field') rather than as typed columns. Reads tolerate missing fields. The relevant question is whether strict consumers downstream of the Parquet (analytics queries, calibration jobs, the separate analytics system) hold tight assumptions about JSON field presence.

For each schema bump (DreamRunEvent v1 → v2, DreamSpecialistEvent v1 → v2, AgentToolSummaryCreatedEvent v1 → v2 — note RepresentationCompletedEvent was already bumped to v2 on this branch and its Phase 4 additions ride along in v2):

  1. Producer-side merge. Bump _schema_version, add fields. Deploy to staging. Xatu accepts everything inside event.data transparently — no Xatu code change required for additive fields.
  2. Verify in staging Parquet. Query the new fields via Athena/DuckDB/whatever queries the S3 archive. Confirm new fields land in the new columns and old events still parse.
  3. Notify downstream consumers of the new fields (calibration jobs, separate analytics) so they can wire up reads. Pin to dataschema URL version to filter old vs new events.
  4. Deploy producer to prod.
  5. Monitor for 24-48 hours. Check honcho_telemetry_events_dropped_total and any Xatu-side ingestion errors.
  6. Consumer cutover. Calibration / dashboards switch from v(N) to v(N+1) columns at their own pace.

For Phase 0 (minimal envelope addition):

  • No per-event _schema_version bump — honcho_version is emitter-injected into the serialized body, not declared on any event class. §4.1 covers this exemption.
  • Verify event.data.honcho_version is present in the JSON-encoded data column via local smoke test: SELECT json_extract(data, '$.honcho_version') FROM events WHERE ....
  • The only emitter-injected body field is honcho_version (Phase 0). Reachable via json_extract(data, '$.honcho_version'). If filtering on it becomes hot, a small Xatu PR can extract it to a top-level Parquet column — not needed v1.

Communication: ping in #telemetry before merge; include link to this PLAN.md and the specific phase.

4.5 Data classification table

A short table everyone (consumers, producers, lint) can point at. Update when adding event fields.

ClassField examplesTreatment
Public scalarmessage_count, total_tokens, duration_ms, iteration, attempt, all counts, all booleans, all enums (outcome, call_purpose)Emit freely.
Resource identifier (low-risk)run_id, tool_call_seq, file_id, message_idHoncho-internal IDs (ULIDs / nanoids / serial); opaque to user. Emit freely.
Resource identifier (sensitive)workspace_name, session_name, peer_name, observer, observed, target_nameAll user-controlled strings — applications choose them and may use real names, email addresses, account IDs. Emit, but document for consumers that these are potentially sensitive. Tenant attribution is server-side at Xatu via Fly app_name, NOT via workspace_name — don’t treat workspace_name as a safe routing key. Consider hashing for any cross-tenant analysis.
External identifierprovider_tool_call_id, message_id, earliest_message_id, latest_message_idEmit. Honcho-internal IDs, opaque to user.
Metadata, case-by-casefilename, content_type, file_size_bytesfilename is borderline — could be PII (john_smith_resume.pdf). Recommend: drop filename from FileUploadedEvent v2, keep content_type/file_size_bytes. If product needs filename for analytics, hash it.
ForbiddenRaw content, text, query, prompt, body, observation_content, summary_textNever emit. PII lint (§5.4.2) enforces.

The PII lint banlist (§5.4.2) covers the “forbidden” row structurally. The “case-by-case” row needs human review at PR time.

4.6 run_id width

  • Today run_id is str(uuid.uuid4())[:8] — 8 chars of the dashed UUID string (so values like "3b1a4c2d" or "3b1a4c2-"; the dash position depends on slice alignment). Effective entropy is ~30 bits since dashes are deterministic at fixed positions and the first 8 chars are just hex.
    • Generated at: src/dreamer/orchestrator.py:92, src/dreamer/specialists.py:156, src/dialectic/core.py:102, src/dialectic/core.py:245.
    • At analytics scale: birthday collision becomes 1% likely around ~9k concurrent runs; at 1M runs collisions are essentially guaranteed. Fine for logs/console; not fine for joins across Athena/BigQuery.
  • Fix: generate a full ULID (128 bits, time-sortable, lexically sortable as strings) at run start. Store as run_id on all events. Keep a separate run_display_id = run_id[:8] for log lines if needed.
  • Migration: producer-side change only — change generation, all events naturally adopt the wider id. Downstream consumers querying run_id as a string column work without change (Parquet stores the JSON value as-is).
  • Checklist (everything that needs updating, not just generation sites):
    1. The four generation sites above.
    2. Event-class docstrings/descriptions in src/telemetry/events/*.py — every run_id: str = Field(..., description="8-char UUID prefix for run correlation") needs to change to “ULID for run correlation”. Currently appears in agent.py, dialectic.py, dream.py.
    3. Test fixtures in tests/telemetry/conftest.py — sample run_id="abc12345" / "def67890" / "ghi11111" are 8-char placeholders. Replace with valid ULIDs (or a deterministic ULID generator for fixtures).
    4. Anywhere in agent prompts or log lines that quotes run_id to the LLM — keep the 8-char run_display_id there for token economy.

4.7 What we are NOT doing

  • No PII leakage: never include raw message content, raw observation content, or raw queries in events. result_chars/result_tokens_estimate are scalars only. Enforced by §5.4.2 PII lint.
  • No observer name hashes for now (req 4): low signal, leaky.
  • No retry / fallback events as a separate type: rolled into LLMCallCompletedEvent.outcome / was_fallback / attempt.
  • No hit_input_token_cap for the deriver until the tool-less LLM path actually enforces max_input_tokens (today it’s silently dropped). See Phase 4 §3.
  • Streamed final-response token totals are not in v1. stream_final_response returns an iterator and we emit a was_stream=True placeholder LLMCallCompletedEvent at setup time. Real token totals require wiring through StreamingResponseWithMetadata.__aiter__ completion — known follow-up. Until then, streamed dialectic answers under-report output tokens in llm.call.completed. The aggregate DialecticCompletedEvent already has the totals — calibration that needs streamed-call accuracy should source from there.
  • Provider-side token estimation differences (OpenAI tiktoken vs. Gemini’s native tokenizer) are accepted; input_tokens_estimate is explicitly an estimate. Don’t try to pull native tokenizer SDKs per provider.

5. Testing strategy

5.1 Unit tests (per phase)

Pattern already established in tests/telemetry/test_events.py: instantiate event, assert event_type(), schema_version(), get_resource_id(), model_dump() shape. Replicate for each new/modified event.

  • Add a fixture per new event in tests/telemetry/conftest.py (mirroring sample_*_event pattern).
  • Update all_sample_events to include new ones — there’s an aggregated test that iterates and validates basics.

5.2 Emitter envelope tests

New test in tests/telemetry/test_emitter.py:

  • Patch settings.TELEMETRY.NAMESPACE and settings.TELEMETRY.HONCHO_VERSION. Emit any event. Inspect the resulting CloudEvent:
    • Envelope: assert source is /honcho/{namespace}/{category} (so Xatu’s namespace-parsing fires). Assert NO honcho* extension attrs are present.
    • Body: assert event.data.honcho_version matches the patched value when set. With the setting None, assert the key is absent from the body (not present-with-null), per the emitter contract in Phase 0.
    • Event-instance non-mutation: capture the event instance before emit(event) and after; assert no fields changed. The emitter must inject into the serialized body only.

5.3 Call-site integration tests

For each emit site, add a focused integration test that:

  1. Spins up the relevant code path with all heavy deps mocked (LLM, embedding, db).
  2. Patches src.telemetry.emitter.get_emitter to return an in-memory recorder.
  3. Asserts the recorded events match expected shape, including new fields.

Sites to add tests for (mostly new):

  • tests/llm/test_telemetry_llm_call.py (phase 1) — assert LLMCallCompletedEvent shape across providers, including fallback path and outcome="error" / is_final_attempt=true for the retry-exhausted case.
  • tests/llm/test_telemetry_iteration.py (phase 2) — call honcho_llm_call with tools + agent_telemetry, assert N iterations → N AgentIterationEvents with correct run_id.
  • tests/utils/test_agent_tools_telemetry.py (phase 3) — call each _handle_*, assert one AgentToolCallCompletedEvent per invocation with right tool_name, tool_call_seq, and search fields populated where relevant.
  • tests/deriver/test_representation_telemetry.py (phase 4) — call process_representation_tasks_batch end-to-end with mocked LLM, assert the new v2-additive fields are correct (use a setup with extra_context_message_count > 0 to exercise the breakdown).
  • tests/dreamer/test_dream_telemetry.py (phase 5).
  • tests/utils/test_summarizer_telemetry.py (phase 6).

5.4 Schema lint

A new test tests/telemetry/test_schema_invariants.py:

  • Iterates all BaseEvent subclasses.
  • Asserts _event_type, _schema_version, _category are set.
  • Asserts get_resource_id() returns non-empty for a sample instance built from defaults+required.
  • Asserts every event class has a BaseEvent.metadata field populated by the emitter at serialization time (envelope-level identity check).

5.4.1 Schema golden tests

For every event class, snapshot event.model_json_schema() to a versioned file under tests/telemetry/golden/{event_type}.v{schema_version}.json. A CI test diffs the live schema against the snapshot — drift fails the build. Bumping _schema_version requires adding a new golden file; modifying an existing version requires deleting and regenerating, which forces a code-review conversation. This catches accidental breaking changes (renamed fields, type changes) that the schema_version bump rule would otherwise miss.

5.4.2 PII lint rule for event payloads

A test that walks every BaseEvent subclass and inspects field names. Banned substrings (case-insensitive): content, text, query (without _tokens/_count suffix), body, message_content, prompt, observation_text. If a field name matches and isn’t on an allowlist of known-safe (e.g. result_tokens_estimate), fail. This is a cheap structural lint that catches the obvious “let me just include the search query” mistake before merge.

Optional follow-up: a runtime emitter assertion in debug builds that round-trips event JSON through the banlist; off in prod.

5.5 End-to-end smoke test

Existing tests/integration/test_telemetry.py exercises the full path with a real-ish emitter. Extend with one new scenario per phase that walks the relevant code path and asserts the expected event types appear in the recorded buffer in order.

5.6 Local manual verification

Add to docs/contributing/telemetry.mdx (new) or update existing telemetry docs:

# Run server with a sink endpoint pointed at a local catcher.
# In staging/prod, TELEMETRY_NAMESPACE must equal the Fly app name so Xatu's
# server-side tenant enrichment fires. Locally any value is fine.
TELEMETRY_ENABLED=true \
TELEMETRY_ENDPOINT=http://localhost:9999/events \
TELEMETRY_NAMESPACE=honcho-local \
TELEMETRY_HONCHO_VERSION="$(uv run python -c 'from importlib.metadata import version; print(version(\"honcho\"))')" \
uv run fastapi dev src/main.py
 
# In another shell, a 5-line FastAPI catcher that prints incoming CloudEvents.
# Inspect event.data.honcho_version on the body.
uv run python scripts/telemetry_catcher.py  # add this script

Then drive the API with a few requests (tests/unified/run.py does this) and inspect that the printed events have the expected new fields. Useful for spot-checking phases 4/5/6 where the data flow is non-trivial.

6. Rollout order & PR strategy

Each phase = one PR, in order. Sizes:

PhaseApprox LOCTouchesRisk
−1~0verification + data hand-off onlyzero
0~50TelemetrySettings.HONCHO_VERSION, emitter.emit body injection, prometheus drop/sample counterslow
1~350llm/, telemetry/events/, runtime.py (AttemptPlan), CallPurpose enum, samplermedium (provider-touching)
2~150llm/, dialectic/, dreamer/low
3~350agent_tools.py (executor wrapper + ToolResult dataclass + ContextVars)medium
4~400deriver/, queue_manager (QueueBatchResult), telemetry/, llm/api.py (tool-less truncation + input_was_truncated on response)medium
5~200dreamer/, queue_payloadlow
6~100summarizerlow
7~250embedding_client.py, agent_tools.py, crud/representation.pymedium

Ship −1 → 0 first (compatibility spike + envelope) — required before any new producer code. Then 1 → 2 (envelope + LLM call + iteration emission). That alone moves the needle for calibration. Then 3 → 7 over the next 2-3 weeks. Phase 7 (embedding) can ship anywhere in the 3-7 window — it doesn’t depend on the others, but it DOES depend on the sampler shipped in Phase 1.

7. Open questions

Resolved (Rev 4 Xatu owner Q&A + Rev 8 spec-owner simplification):

  • Tenant model → Xatu resolves server-side via app_name. Honcho-side never sets a tenant_id. workspace_name stays a data-level field, NOT a tenant key. TELEMETRY_NAMESPACE = <fly_app_name> is already correctly set in every deployed env per spec owner.
  • CloudEvents batch + dedupe → Tenant-scoped Redis SET NX with 48h TTL. Already working in prod. Fail-open during Redis outage; defense-in-depth dedupe on GROUP BY event.id in analytics queries.
  • Tool-less truncation in api.py → Promoted into Phase 4. No longer an open question.
  • Envelope metadata expansion → Reduced to honcho_version only. No git_sha / trace_id / environment / envelope_version / EventMetadata nested object. Xatu PR for typed metadata columns is moot (no metadata object to extract).

Still open:

  1. Sample-rate sizing: default HIGH_VOLUME_SAMPLE_RATE = 0.1 was chosen heuristically. Needs to be reconciled against the $/month projection from Xatu owner (Phase −1 step 3). Production rate may be higher or lower.
  2. LiteLLM model-string parity: does Honcho’s model field match what LiteLLM bills? Verify with a spot-check after Phase 1 ships.
  3. ClickHouse — does it exist elsewhere? Xatu uses S3 Parquet, not ClickHouse. The “separate analytics system” mentioned in user memory may be ClickHouse-based — if so, that’s a separate downstream coordination, not Xatu’s problem. Identify the owner and confirm migration sequence.
  4. git_sha (deferred, not blocking): if a deploy-forensics question comes up later — “which deploy emitted this event when we had two revs of the same honcho_version in flight” — adding git_sha is ~3 LOC (env var + body field). Easy to revisit.

8. Quick reference: instrumentation insertion points

ConcernFileSymbol
LLMCallCompletedEvent (success + failure)src/llm/executor.pyhoncho_llm_call_inner (try/finally); reads LLMTelemetryContext kwarg
Iteration emit (per LLM response)src/llm/tool_loop.pyimmediately after response = await call_func() (~line 249), before the no-tool return
Iteration emit (synthesis call)src/llm/tool_loop.pyafter final_response = await final_call_func() (~line 470)
iteration in LLMTelemetryContextsrc/llm/tool_loop.pyset before each honcho_llm_call_inner invocation — NOT via the existing set_current_iteration ContextVar (that fires after)
AgentToolCallCompletedEvent emitsrc/utils/agent_tools.pyinner execute_tool closure in create_tool_executor; reads tool_call_seq ContextVar
tool_call_seq + provider_tool_call_idsrc/llm/tool_loop.pyset before invoking tool_executor in the per-iteration loop
Tool handler metadata (for specialist rollups)src/utils/agent_tools.pyeach _handle_* returns ToolResult(content, metadata); loop puts it on all_tool_calls[i]["tool_result_metadata"]
Representation v2 additive fieldssrc/deriver/deriver.pyprocess_representation_tasks_batch
Observer fanoutsrc/deriver/deriver.pysame
Batch cap-hit flagssrc/deriver/queue_manager.pyget_queue_item_batch returns QueueBatchResult
Dream trigger reasons (threshold_reason, delay_reason)src/dreamer/dream_scheduler.pycheck_and_schedule_dream (pass via dream payload)
Dream specialist rollupssrc/dreamer/specialists.pySpecialist.run — sums from tool_result_metadata (NOT name counts)
Summary v2 fieldssrc/utils/summarizer.pycomputed in summarize_if_needed, passed into _create_summary
EmbeddingCallCompletedEventsrc/embedding_client.pyeach provider call site
embedding_call_purpose ContextVarsrc/utils/types.py + caller siteswrap embedding-driving operations in embedding_call_purpose("…"): context manager
honcho_version body injectionsrc/telemetry/emitter.pyemit adds body["honcho_version"] from settings.TELEMETRY.HONCHO_VERSION; event instance untouched; NOT a CloudEvent extension attr
Fly app name → tenant routingdeployment configalready set per spec owner — TELEMETRY_NAMESPACE = <fly_app_name>. Verify only.
Emitter health metricssrc/telemetry/emitter.pyemit (drop on buffer overflow), sampler entry (intentional sample-out), _send_batch (send failure)
run_id generationsrc/dreamer/orchestrator.py:92, src/dreamer/specialists.py:156, src/dialectic/core.py:102,245switch str(uuid.uuid4())[:8] to full ULID; keep 8-char display id for log lines
Service identitysrc/config.py::TelemetrySettingsone env-driven field: HONCHO_VERSION (with importlib.metadata fallback)

Graph View