Plastic Labs Internal Wiki

Recently Updated

Honcho instance debugging

4 min read

One tenant’s managed Honcho instance is misbehaving and you have already narrowed it to that tenant. If you have not — if you are starting from an alert — start at Tenant debugging, which covers finding the tenant and reading exit reasons, then come back here.

Each tenant is two Deployments, app-api and app-deriver, in a namespace named cluster-{N}-batch-{M}-{app_name}, with its own database on one of the AlloyDB clusters. Both containers run the same Honcho image.

What state does Groudon think it is in?

Start in the orchestrator database, not the cluster. Connection details are in connecting to the orchestrator DB.

SELECT t.name, t.instance_state, hi.app_name, hi.version, hi.active,
       td.id AS tenant_database, tba.cluster_index, tba.batch_index
FROM groudon.honcho_instances hi
JOIN groudon.tenants t ON t.id = hi.tenant_id
LEFT JOIN groudon.tenant_databases td ON td.id = hi.tenant_database_id
LEFT JOIN groudon.tenant_batch_allocations tba ON tba.tenant_id = t.id
WHERE hi.app_name = 'hch...';

Read instance_state before anything else:

  • live — a latch. It proves the transition to serving happened, not that the instance is serving now. A live tenant with no pods is possible; look at the cluster before assuming the pipeline is stuck.
  • cold_storage — deliberately at replicas: 0. No pods is correct. Traffic brings it back automatically.
  • updating — a version bump was recorded and the rollout has not been observed landing yet. Normal for a few minutes; see Releasing a Honcho version.
  • pending / errored — provisioning did not finish. The repair maintenance job retries these every 3 minutes.
  • failed — terminal. An upgrade that could not recover, or an IntegrityError during creation. Not retried automatically; needs manual repair.

Is the deriver doing work?

The queue lives in the tenant’s own database, not the orchestrator. The fastest read on whether a deriver is productive:

SELECT task_type, processed, count(*), min(created_at), max(created_at)
FROM queue GROUP BY 1, 2 ORDER BY 1, 2;
 
-- work units currently claimed
SELECT work_unit_key, last_updated FROM active_queue_sessions
ORDER BY last_updated DESC LIMIT 20;
 
-- anything that errored
SELECT id, task_type, error, created_at FROM queue
WHERE error IS NOT NULL ORDER BY created_at DESC LIMIT 20;

A growing processed = false backlog with a stale max(created_at) in active_queue_sessions means the deriver is not picking work up. A backlog with fresh claims means it is working and simply behind.

Pods and logs

C=gke_plastic-labs-prod_us-east4_cluster-33
NS=cluster-33-batch-4-hch...
 
kubectl --context $C -n $NS get pods
kubectl --context $C -n $NS logs deploy/app-deriver --tail=100
kubectl --context $C -n $NS logs deploy/app-api --previous --tail=40

--previous is the container that died — without it you get the current retry, which has not failed yet. Exit reasons are decoded in Tenant debugging.

If the pod is already gone, kubectl logs has nothing. Vector ships from every node to vector-aggregator in the operations cluster, which sinks to Pub/Sub and BigQuery. That is the historical path.

A 1/1 Running pod can still be dead to the network — probes are localhost-only. See Node loses egress.

Config and secrets

Both containers get their config from Secret/honcho-secrets in the namespace, which External Secrets syncs from AWS Secrets Manager, plus explicit env entries in the templates that override it (an explicit env beats envFrom). Notable ones set that way on the deriver: CACHE_ENABLED=false, VECTOR_STORE_MIGRATED=true, CACHE_URL pointing at redis.prod.internal:6379.

kubectl --context $C -n $NS get secret honcho-secrets \
  -o jsonpath='{.data.MODEL_PROVIDER}' | base64 -d
 
kubectl --context $C -n $NS get externalsecret honcho-secret -o yaml | tail -20

A running pod never sees a changed secret. Rotating one means restarting pods (roll-secret.sh, not a bare rollout restart).

Per-tenant overrides come from secret_overrides in Groudon and are merged into the same Secret by the ExternalSecret template — so a bad override shows up here, not in AWS.

Metrics

honcho_endpoint_counts_total carries per-tenant request counts by resource and status_code. Query Mimir through Grafana, or directly — see request rates, latency, error codes. Per-tenant latency and error rates are also on the Groudon dashboard’s metrics page.

Vector store

Honcho uses Turbopuffer as its vector store. Namespaces are {prefix}.{type}.{hash}, where prefix is VECTOR_STORE_NAMESPACE, type is doc (observations) or msg (messages), and hash is a 43-char base64url SHA-256. Document namespaces hash "{workspace_name}.{observer_name}.{observed_name}"; message namespaces hash just "{workspace_name}". The inputs are peer and workspace names, not IDs. To reproduce a hash, use the hashing method in src/vector_store/__init__.py.

See also

Graph View

Backlinks