Platform knowledge transfer

How the supporting platform pieces work — billing, observability, caching, secrets, CI — and what to check when each one misbehaves. This is background knowledge; for “something is broken right now” go to Where to look.

Most of what is below runs on GKE. Fly.io still hosts the CI runners (Ivysaur), leftover Grafana dashboards (prod-grafana), and the Tentacruel LiteLLM proxy. If a command here says kubectl, that is not a translation of an older fly command; that workload genuinely moved. fly logs -a ... is only useful for the Fly leftovers.

Billing and usage metering (Xatu)

Xatu is a standalone telemetry service that ingests metrics from Honcho for 3.x+ instances. It consumes CloudEvents, writes them to S3 as parquet, and creates Stripe meter events for the billing-relevant ones (currently dialectic.completed and representation.completed).

It runs as two Deployments in the xatu namespace on the operations cluster, in both plastic-labs-staging and plastic-labs-prod.

xatu-ingestion — reachable in-VPC at xatu-ingestion.xatu.prod.internal (the value of TELEMETRY_ENDPOINT on every Honcho pod):

• consumes CloudEvents from Honcho
• deduplicates on event_id via Redis
• publishes to the managed Redpanda topic honcho-events, keyed by tenant_id so a tenant’s events stay grouped and ordered

xatu-consumer:

• polls the topic, buffers in memory, periodically flushes to S3 as parquet and reports usage to Stripe
• uses Stripe’s Meter Event Stream API with event IDs as the idempotency key
• commits Redpanda offsets only after the S3 write and Stripe report succeed

Legacy 2.x instances still report usage through Groudon’s passthrough method.

Pulse check

• Consumer lag on honcho-events in the Redpanda console. Should be at or near zero. Growing lag means the consumer is stuck or falling behind.
• The CloudEvents Grafana dashboard for event flow — it queries the S3 parquet through the motherduck-duckdb-datasource plugin, so it shows event volume by type across all tenants.
• Logs:

  kubectl --context gke_plastic-labs-prod_us-east4_operations -n xatu \
    logs -l app=xatu-consumer --tail=100

Xatu is stable but it is on the revenue path, so it is worth a look a few times a day.

Observability

Live metrics and logs terminate on the operations cluster. Leftover Grafana dashboards still exist on Fly (prod-grafana); sableye is the historical origin of the scrape config, not a running deployment.

Metrics — every cluster runs a vmagent (the observability-agent chart) that remote_writes to Mimir in the metrics namespace on operations. Mimir is a single-instance monolith; the authoritative blocks live in a regional GCS bucket and the PersistentVolume holds only the WAL and compactor staging. A separate vmagent-external scrapes third-party endpoints (Redpanda, Modal, and other managed services) — the job that sableye/config/prometheus.yml used to do.

Logs — every node runs a Vector agent shipping to vector-aggregator in the log-ingest namespace on operations, which sinks to Pub/Sub and on to BigQuery.

Grafana is in-cluster too, in the metrics namespace, pre-provisioned with Mimir as its default datasource, behind an internal load balancer with Google SSO. URLs are in Where to look; querying Mimir directly is covered there as well.

Caching (Redis)

Memorystore Redis Cluster, not Upstash. Two clusters per environment:

• plastic-labs-prod-redis-cluster (3 shards) at redis.prod.internal:6379 — Honcho + Groudon
• xatu-telemetry-production at redis.telemetry.prod.internal:6379 — Xatu event dedup

What it holds:

• Resource caching (workspace, peers, sessions) in Honcho — API only; CACHE_ENABLED is explicitly false on derivers, because each pod holds roughly one connection per shard and derivers are half the pods while gaining least from the cache
• Whether a tenant’s derivers are paused for $0 Stripe balance (Groudon)
• API key cache invalidation across gateway pods (Groudon)
• Event deduplication during billing ingestion (Xatu)

It reaches the VPC over Private Service Connect, so it holds an ordinary 10.127.x address in the operations subnet.

gcloud redis clusters list --project=plastic-labs-prod --region=us-east4 \
  --format="table(name,shardCount,discoveryEndpoints[0].address,state)"

Connect from inside the VPC — a pod on the operations cluster, or a port-forward. It is not reachable from a laptop.

A dead Redis does not fail a health check. Xatu’s /health returns healthy unconditionally and its dedup swallows every exception, so pods stay 1/1 Running with deduplication silently disabled.

Secrets

Secrets live in AWS Secrets Manager, as a JSON blob per Honcho version: honcho/secrets/<environment>/<version> (e.g. honcho/secrets/honcho-production-saas/3.1.0). Tenant database credentials are separate, under honcho/tenant-databases/{stage}/{id}.

They reach pods through the External Secrets Operator, which runs on every cluster and syncs each namespace’s ExternalSecret into Secret/honcho-secrets on a 1-hour refreshInterval. Pods read it with envFrom, evaluated only at container start — so rotating a value requires restarting pods. Use roll-secret.sh, not a bare rollout restart.

load_honcho_secrets in Groudon writes new versions; see Releasing a Honcho version onto Groudon.

Access paths:

• GitHub Actions assume the GitHubActionsS3Role IAM role.
• Groudon reads via the groudon-secrets-reader IAM user.
• ESO authenticates as its own principal against the aws-sm ClusterSecretStore.

Local use needs AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION=us-east-1 from the honcho-production-saas 1Password.

Databases

Tenant databases are AlloyDB, roughly 20 HA clusters, each hosting many tenant databases named after the instance’s app_name. Groudon’s own orchestrator state is a separate AlloyDB cluster. Saturation and connection diagnostics are in AlloyDB connection saturation.

Supabase and the Supavisor pooler are gone. The old pooler monitoring is kept only for historical incidents — legacy.

Vector store (Turbopuffer)

Turbopuffer is Honcho’s vector store, and the pgvector dual-write migration is complete — VECTOR_STORE_MIGRATED is pinned true in the templates, so Turbopuffer is the source of truth for both reads and writes.

Namespace format is {prefix}.{type}.{hash}: prefix from VECTOR_STORE_NAMESPACE, type is doc (observations) or msg (messages), hash is a 43-char base64url SHA-256. Document namespaces hash "{workspace_name}.{observer_name}.{observed_name}"; message namespaces hash just "{workspace_name}". Inputs are peer/workspace names, not IDs. To reproduce one, use the hashing method in src/vector_store/__init__.py.

CI test runners

This is the one place Fly.io is still load-bearing. Ivysaur deploys ephemeral self-hosted GitHub Actions runners on Fly for Honcho and Groudon test suites. It pre-pulls test images and pre-installs Playwright browsers, registers the machine as a runner, and destroys it when the job finishes. Configured in test-configs.json:

• honcho-unified-runner — pre-pulls pgvector/pgvector:pg15 and redis:8.2-alpine
• groudon-e2e-runner — pre-pulls the Supabase stack, plus Playwright Chromium and Firefox

Honcho unified tests (.github/workflows/unified-tests.yml) run on push to main when src/** or tests/** change, and on a PR labelled run-unified-tests (code owners only; the label is purged as the run starts so it can be re-added). They start a full HonchoHarness — Postgres, Redis, FastAPI, deriver via Docker Compose — and run ~33 JSON-defined cases from tests/unified/test_cases/. Results go to S3 and the Discord devops channel.

Two gotchas worth knowing, because they cost a day:

• The gate job only runs on pull_request: labeled, so it is skipped on a push to main. Downstream jobs need !cancelled() (not a plain success() dependency) or the whole suite silently skips on merge.
• A queued run that “cannot get a runner” usually means the Fly machine failed to register, not that the suite is broken. Check the start-runner job.

Groudon E2E (.github/workflows/e2e-tests.yml) runs Playwright against the dashboard on push to main / stable. Historically flaky.

Sentry and alerting

Sentry projects: Honcho, Groudon, Groudon-web, Xatu. Alerts route through Rootly, which is also where the public status page is driven from — see Incident response.

Sentry is noisy by default. Two things that generate false pages and are worth recognising:

• Spot preemptions. Prod derivers run on spot nodes and a preempted pod’s shutdown path can look like an error. See Compute classes and spot.
• Rollout churn. A fleet-wide version bump recreates thousands of pods; transient startup errors during that window are expected.

See also

• Where to look — consoles, credentials, the signals that lie
• ArgoCD operations
• Honcho instance debugging
• Infra — the full topology this page’s pieces sit in
• Load testing