This spec covers the instrumentation side of Honcho’s telemetry: what to do with Sentry, Prometheus, and the scattered accumulate_metric calls. The event/analytics side — CloudEvents emission, the Xatu pipeline, Parquet on S3, and DuckDB/ClickHouse querying — is covered separately in CloudEvents pipeline and is not repeated here.
Current instrumentation systems
| System | Purpose | Location | Notes |
|---|---|---|---|
| Sentry | Tracing + errors | src/sentry.py, src/utils/tracing.py | @with_sentry_transaction decorator, DSN-configured |
| Prometheus | Pull-based metrics | src/prometheus.py | Counters for API requests, tokens, queue items |
accumulate_metric | Local metric collection | src/utils/logging.py | In-memory dict, optional file export |
| MetricsCollector | Benchmark aggregation | src/utils/metrics_collector.py | JSON export, cross-process via file locking |
| Langfuse | LLM tracing | src/utils/logging.py | Conditional @observe decorator |
Existing Prometheus counters: API_REQUESTS, MESSAGES_CREATED, DIALECTIC_CALLS, DERIVER_QUEUE_ITEMS, DERIVER_TOKENS, DIALECTIC_TOKENS, DREAMER_TOKENS.
Problems
- Pull-based Prometheus is fine for operations but wrong for billing — scrape intervals miss events, counters lack event-level granularity, and tenant attribution stops at the namespace label.
accumulate_metricis scattered — in-memory (lost on restart), file-based collection is process-local, no schema.- Sentry coupling — great for errors, awkward and not cost-effective for high-volume custom analytics.
Direction: three lanes, clear boundaries
Each telemetry concern gets one home:
- CloudEvents → business events (billing, analytics, audit). See CloudEvents pipeline.
- OTel metrics → operational telemetry (latencies, queue depth, request rates), pushed to Mimir.
- Sentry → incident response (errors, alerting). Keep it.
Sentry
Recommendation: keep Sentry for errors and alerting. CloudEvents are not designed for distributed tracing, and Sentry’s error workflow is worth keeping. OTel can optionally take over traces over time, but Sentry stays the incident-response surface.
Prometheus → OTel metrics (push to Mimir)
Migrate from scrape-based Prometheus to push-based OTel metrics via Prometheus Remote Write into Mimir. This removes the scrape-target/vmagent complexity that ephemeral Fly VMs make painful.
Current: Honcho /metrics ──► Fly scrape ──► vmagent ──► S3 + Mimir
Proposed: Honcho (OTel SDK) ──► Prometheus Remote Write ──► Mimir
Mimir accepts Remote Write at /api/v1/push; the OTel SDK pushes directly via opentelemetry-exporter-prometheus-remote-write (with X-Scope-OrgID as the Mimir tenant header). Benefits: no scrape targets to manage, no vmagent duplication, same Grafana/PromQL dashboards, built-in retry/backoff.
Where the current counters go:
| Current counter | Category | New home |
|---|---|---|
api_requests_total | Operational | OTel Counter |
messages_created_total | Resource event | CloudEvents |
dialectic_calls_total | Activity event | CloudEvents + OTel Counter |
deriver_queue_items_processed_total | Operational | OTel Counter |
deriver_tokens_processed_total | Work event | CloudEvents + OTel Histogram |
dialectic_tokens_processed_total | Activity event | In the dialectic CloudEvent |
dreamer_tokens_processed_total | Work event | CloudEvents + OTel Histogram |
OTel/CloudEvents overlap is intentional: OTel for real-time dashboards, CloudEvents for queryable history. Deprecate Prometheus counters last.
accumulate_metric
Recommendation: keep it for local development and debugging, but production billing should emit events directly from each code path rather than relying on accumulated in-memory metrics. Don’t bridge it into the billing pipeline.
accumulate_metric audit
Classification of every existing accumulate_metric call, by destination. Legend:
- CloudEvents — billing-critical; needs idempotency, tenant attribution, long-term storage
- OTel — aggregated operational metrics for dashboards/alerting
- Structured log — debug/error context, ephemeral
- Remove — redundant or derivable
src/deriver/deriver.py
| Metric | Destination | Reasoning |
|---|---|---|
starting_message_id / ending_message_id | Structured log | Debug correlation |
context_preparation, llm_call_duration, total_processing_time | OTel Histogram | Latency |
observation_count | OTel Counter | Obs/task |
messages, explicit_observations | Structured log | Debug blobs (gated on LOG_OBSERVATIONS) |
Token counts here (via DERIVER_TOKENS_PROCESSED) should become a CloudEvents TokensProcessedEvent.
src/crud/representation.py
| Metric | Destination |
|---|---|
embed_new_observations (ms) | OTel Histogram (embedding latency) |
save_new_observations (ms) | OTel Histogram (DB write latency) |
src/dialectic/core.py
| Metric | Destination | Reasoning |
|---|---|---|
context, query, prefetched_observations, thinking, response | Structured log | Debug blobs |
tool_calls | CloudEvents | In DialecticCompletedEvent |
total_duration | CloudEvents + OTel | Event + histogram for SLA |
input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens | CloudEvents | In DialecticCompletedEvent |
uncached_input_tokens | Remove | Derivable from other token fields |
All dialectic token/timing metrics consolidate into a single DialecticCompletedEvent.
src/dreamer/orchestrator.py
| Metric | Destination |
|---|---|
surprisal_observations | OTel Counter |
surprisal_error, deduction_result/deduction_error, induction_result/induction_error | Structured log |
total_duration | OTel Histogram |
src/dreamer/specialists.py
| Metric | Destination |
|---|---|
total_duration | OTel Histogram |
tool_calls | OTel Counter |
input_tokens, output_tokens | CloudEvents (billing — dreamer token usage) |
src/utils/summarizer.py
| Metric | Destination |
|---|---|
*_summary_up_to_message, *_summary_text | Structured log |
*_summary_size (chars) | OTel Gauge |
*_summary_creation (ms) | OTel Histogram |
Summarizer token usage (via DERIVER_TOKENS_PROCESSED) should become CloudEvents.
What becomes OTel metrics
# Latency histograms
deriver_context_prep_duration = meter.create_histogram("deriver_context_prep_seconds")
deriver_llm_call_duration = meter.create_histogram("deriver_llm_call_seconds")
deriver_total_duration = meter.create_histogram("deriver_total_seconds")
dialectic_total_duration = meter.create_histogram("dialectic_total_seconds")
dreamer_total_duration = meter.create_histogram("dreamer_total_seconds")
summarizer_duration = meter.create_histogram("summarizer_seconds")
embedding_duration = meter.create_histogram("embedding_seconds")
db_write_duration = meter.create_histogram("db_write_seconds")
# Counters
observations_created = meter.create_counter("observations_created_total")
tool_calls = meter.create_counter("tool_calls_total")
# Gauges
summary_size = meter.create_gauge("summary_size_chars")What becomes structured logs
All blob-type metrics (raw context, queries, thinking traces, responses) become structured log calls rather than accumulated metrics.
Open questions
- Idempotency level: per-task for the deriver (natural boundary), per-batch for message creation, per-call for dialectic.
accumulate_metricfate: keep for dev/debug; production billing emits events directly.- Prometheus fate: migrate operational counters to OTel over time; billing counters move to CloudEvents.