Incident Response & On-Call

How we run incident response in Rootly: the service topology, the public status page, and the on-call rotation. This is the source of truth; the Linear project Rootly — Incident Management & On-Call tracks the setup work and links here.

Model

Rootly has three catalogs, and they’re easy to conflate:

• Services — internal infra components. Alerts route here.
• Functionalities — customer-observable capabilities. These feed the status page.
• Environments — deploy stage (Production / Staging).

Many services roll up into one functionality. Only Production pages.

Topology

flowchart LR
  subgraph SRC[Alert sources · v1]
    SEN[Sentry]
    GCP[GCP monitoring + uptime]
    GRA[Grafana / Prometheus]
  end

  subgraph SVC[Rootly Services · internal]
    GW[Groudon Gateway]
    GH[Groudon Health]
    GA[Groudon Admin]
    GD[Groudon Dashboard]
    HA[Honcho API]
    HD[Honcho Deriver]
    XI[Xatu Ingestion]
    XC[Xatu Consumer]
    SA[Supabase Auth]
    TC[Tentacruel · LiteLLM]
    MG[Metagross · vLLM]
  end

  subgraph FUN[Functionalities → status page]
    F1[REST API]
    F2[Memory processing]
    F3[Recall / Dialectic]
    F4[Dashboard]
    F5[Website & Docs]
    F6[Authentication]
    F7[Billing & usage]
  end

  SRC --> ROUTE{{Alert routing · env=Production}}
  ROUTE --> EP[Escalation: Primary → Secondary → all-Platform]
  ROUTE -.tags service.-> SVC

  GW --> F1
  HA --> F1
  HA --> F3
  HD --> F2
  GD --> F4
  SA --> F6
  XI --> F7
  XC --> F7

Environments

Grounded in groudon/.env (APP_ENV = development | staging | honcho-production-saas).

• Production — required, red; the only environment that pages. Gate every paging workflow on Environment = Production.
• Staging — non-paging (Slack only).
• Development is not modeled — don’t create environments you don’t operate distinctly.

Services (internal catalog)

Service	Repo / process	Alert sources	Notes
Groudon Gateway	groudon · gateway :8000	GCP uptime, Sentry, Grafana	Public API routing by key
Groudon Health	groudon · health :9090	Grafana, Sentry	Monitoring, recovery, upgrades, placeholder pool
Groudon Admin	admin :8001	Sentry	Internal tenant/user mgmt
Groudon Dashboard	dashboard (Next.js, app.honcho.dev)	GCP uptime, Sentry	Vercel
Honcho API	honcho · api :8000	GCP uptime, Sentry, Grafana	REST + dialectic
Honcho Deriver	honcho · deriver	Grafana, Sentry	deriver + dreamer background agents
Xatu Ingestion	xatu · ingestion :9000	Sentry, Grafana	CloudEvents intake
Xatu Consumer	xatu · consumer	Sentry	Redpanda → S3/Stripe
AlloyDB (control plane)	managed	Grafana / GCP	Groudon groudon schema
AlloyDB (tenant)	managed	Grafana / GCP	One database per Honcho instance
Supabase Auth	managed	manual	Dashboard auth (JWT, MFA)
Tentacruel	platform/tentacruel	Sentry, Grafana	LiteLLM proxy — LLM routing
Metagross	platform/metagross	Grafana	vLLM inference (tinybox)

Per-tenant modeling. “One tenant = one Honcho instance” is ~22,300 instances across 46 GKE clusters. Do not create a Rootly service per tenant. Keep Honcho API and Honcho Deriver as one logical service each; carry cluster and namespace as alert fields so routing and filtering still work per-tenant.

Public status page

status.honcho.dev, Honcho Cloud only — side-products (Kyogre etc.) stay off. 7 components in 3 groups + an informational providers group.

Group	Component	Backing services	Health signal
API	REST API (api.honcho.dev)	Groudon Gateway + Honcho API	auto — GCP uptime
	Memory processing	Honcho Deriver	manual → Grafana deriver-queue-depth probe
	Recall / Dialectic	Honcho API (dialectic)	manual → deep uptime check on dialectic endpoint
Web	Dashboard (app.honcho.dev)	Groudon Dashboard	auto — GCP uptime
	Website & Docs (honcho.dev)	Vercel / Cloudflare	uptime check TBD — add to M2
Account	Authentication	Supabase Auth	manual (or link Supabase status)
	Billing & usage	Xatu	manual
Model providers (info)	—	Anthropic / OpenAI / Vertex	informational — link provider status pages

Ship manual signals as manual-during-incident and add probes as they come online — don’t block launch on them.

On-call

One combined weekly on-call rotation — one hat, dev-team-staffed (decided 2026-07-23). The weekly owner carries both jobs; split into two hats only if volume outgrows one. (“Rotation” here means the org duty, not a Rootly object — see Schedules vs. rotations below.)

• Incidents (paging) — prod alerts from Sentry / GCP / Grafana, and postmortems. Real paging.
• Support & OSS (non-paging) — Discord + email Q&A, OSS issue/PR triage, contributor Q&A. Run the pr-triage skill at handoff. First-response + routing; deep code review routes to the area reviewer, not the on-call person.

Schedules vs. rotations (Rootly terms). A schedule is an on-call role that resolves to exactly one person at any moment (it’s what the escalation policy pages); a rotation is the rule inside a schedule that fills it — member order + cadence + handoff. One hat ≠ one schedule: our single combined hat is built as two schedules, Platform Primary and Platform Secondary, because a primary plus a live backup must be two schedules (one schedule = one current person). Each has a single weekly rotation over the same dev-team list, offset a week so this week’s Secondary is next week’s Primary. The escalation policy pages Primary → Secondary → all-Platform. (Multiple rotations within one schedule are only for splitting one seat across time — e.g. follow-the-sun — which we don’t need yet.)

Escalation vs. notification. The escalation policy escalates the person (Primary → 5m → Secondary → 5m → all-Platform). Each person’s notification rules escalate the medium (push 0m → SMS 5m → phone 10m). Keep loudness in notification rules, not duplicated across policy levels.

Alert sources (v1): Sentry, GCP monitoring/uptime, Grafana. Fly.io alerting is out of scope for v1.

Intake funnel: Discord/email + OSS GitHub issues/PRs → Linear Triage → answer/close or convert to a routed pillar issue (type: bug|feature).