Operational procedures — how to run, release, and debug things in production.
On-call
- Where to look — something is wrong and you do not know where yet. Consoles, how to reach them, signals that lie.
- Tenant debugging — start here for most alerts. Finding a tenant, reading exit reasons, and telling a tenant problem from a wider one.
- Honcho instance debugging — one tenant’s instance: state, deriver queue, config, etc.
- Node loses egress — a node stops reaching the internet while its pods stay
Runningand pass every probe. - AlloyDB connection saturation — “remaining connection slots are reserved”, why per-database pooling causes it, and the
max_connectionsfix.
”Why has my change not landed”
- ArgoCD operations — upgrading the control plane. ArgoCD is a Helm release; it does not manage itself.
Support
- Support triage — a user writes in. Email → tenant → verdict; the welcome-credit decision table; pause and balance-email behavior.
- Stripe debugging — the Stripe side: find a tenant’s customer, check and grant credits, invoices and the paid-invoice reissue, coupon codes.
Operations
- Honcho release process — cutting a Honcho version (changelog, tag, staging).
- Releasing a Honcho version onto Groudon — secrets, registering the version, rolling the fleet, auto-upgrade.
Legacy
- Connection-saturation monitoring (Supavisor) — the shared-Postgres pooler monitor, kept for older incidents and for
lock_monitoring.sql.