Plastic Labs Internal Wiki

Recently Updated

Groudon Architecture

14 min read

Groudon is the multi-tenant management platform for Honcho. It provisions dedicated Honcho instances on GKE, routes inbound API traffic by API key, meters usage through Stripe, and monitors instance health. Groudon is the only thing that sits in front of a tenant’s Honcho.

This doc covers the services and the hot paths. How the tenant workloads (the Honcho API and deriver) are deployed is the GitOps pipeline. Pricing is Platform Pricing and Costs.

What this is

The vocabulary, because the rest of this page assumes it.

  • A tenant is a customer organization. One row in tenants. The row is the tenant even before anything is running.
  • An instance is that tenant’s private Honcho: an app-api Deployment, an app-deriver Deployment, and its own AlloyDB database. One row in honcho_instances. A Deployment is Kubernetes’ way of keeping N copies of a process running.
  • An app_name is the instance’s identifier everywhere, and is also the database name within an AlloyDB cluster, a hch-prefixed nanoid like hch4wciennr9itqgxao3gkwa. It appears in the namespace name, the routing hostname, and the secret paths.
  • A cluster is a GKE Autopilot cluster (cluster-0 through cluster-45), plus one operations cluster that runs Groudon itself. Autopilot means Google manages the nodes.
  • A batch is a group of tenants inside a cluster. It exists so ArgoCD has a unit of work smaller than “all 500 tenants on this cluster.”
  • An instance_state is where the tenant is in its lifecycle. The Groudon Gateway reads it on every request to decide whether to route, reject, or wake.
  • A placeholder is an already-provisioned instance with no real tenant yet. Claiming one is how signup skips a cold start.

A tenant’s namespace is cluster-{c}-batch-{b}-{app_name}, which encodes all three placements in one string. When you have only a namespace name, you know the cluster, the batch, and the instance.

Scale

(Numbers as of 2026-08-20)

Tenant clusters46 (cluster-0 … cluster-45)
Tenants per cluster500 (50 batches × 10 tenants)
Allocated instances~22,300
PostgresAlloyDB, ~20 High Availability clusters, one database per instance

High-level topology

End users hit Groudon, which fans out to per-tenant Honcho instances spread across the tenant clusters. Each instance has its own database.

flowchart TB
    Client["End User / Agent SDK"]
    Admin["Tenant Admin (browser)"]

    subgraph Ops["operations cluster"]
        GW["Groudon Gateway :8000"]
        HL["Health :9090"]
        AD["Admin :8001"]
        ACD["ArgoCD"]
    end

    Dash["Dashboard (Vercel)"]

    subgraph C0["cluster-0"]
        EP0["edge-proxy"]
        T0A["tenant A<br/>api + deriver"]
        T0B["tenant B<br/>api + deriver"]
    end

    subgraph C45["cluster-45"]
        EP45["edge-proxy"]
        T45["tenant N<br/>api + deriver"]
    end

    subgraph Data["Data"]
        CP[("AlloyDB<br/>groudon schema")]
        TD[("AlloyDB<br/>one DB per instance")]
    end

    Client -->|API key| GW
    Admin --> Dash --> GW

    GW -->|"JWT + Host header"| EP0
    GW -->|"JWT + Host header"| EP45
    EP0 --> T0A
    EP0 --> T0B
    EP45 --> T45

    GW --> CP
    HL --> CP
    AD --> CP
    ACD -->|sync| C0
    ACD -->|sync| C45

    T0A --> TD
    T0B --> TD
    T45 --> TD

Groudon never reads tenant data. Workspaces, peers, sessions, messages, and observations live in each tenant’s own database. Groudon only routes.

The diagram above is where traffic goes. The next one is the same system by service, including billing and the GitOps handoff.

Service topology

Five services: Groudon Gateway, Admin, Health, Dashboard, and Xatu. Gateway, Admin, and Health share the groudon/ core. The dashboard is a Next.js app on Vercel. Xatu is a separate FastAPI service on the operations cluster.

flowchart TB
    subgraph Clients["Clients"]
        EndUser["End User / Agent SDK"]
        TenantAdmin["Tenant Admin (Browser)"]
        Internal["Internal Operator"]
    end

    subgraph Groudon["Groudon Control Plane (operations cluster)"]
        Gateway["Groudon Gateway :8000<br/>passthrough + dashboard API<br/>webhooks + billing controls"]
        Admin["Admin :8001<br/>internal tenant/user mgmt"]
        Health["Health :9090<br/>repair · upgrade · placeholder pool<br/>Prometheus metrics"]
        Dashboard["Dashboard :3000<br/>Next.js 15 self-service UI"]
    end

    subgraph Honcho["Per-Tenant Honcho Instances (GKE)"]
        EP["edge-proxy<br/>one per tenant cluster"]
        H1["Tenant A namespace<br/>app-api + app-deriver"]
        H2["Tenant B namespace<br/>app-api + app-deriver"]
        Hn["Tenant N namespace<br/>..."]
    end

    subgraph Xatu["Xatu Billing Pipeline (operations cluster)"]
        XIngest["Ingestion<br/>POST /v1/events"]
        Redpanda["Redpanda<br/>topic: honcho-events"]
        XConsumer["Consumer<br/>batch → Stripe + S3"]
    end

    subgraph GitOps["GitOps pipeline"]
        HR["groudon-health render → ArgoCD"]
    end

    subgraph External["External Services"]
        AlloyDB[("AlloyDB<br/>control plane + per-tenant")]
        Supabase[("Supabase Auth")]
        Stripe["Stripe"]
        Redis[("Memorystore<br/>cache + pause state")]
        Resend["Resend (email)"]
        PostHog["PostHog"]
        Sentry["Sentry"]
        S3[("S3<br/>Parquet archive")]
    end

    EndUser -->|API key| Gateway
    TenantAdmin --> Dashboard
    Dashboard -->|/dashboard/v1/*| Gateway
    Internal --> Admin

    Gateway -->|"JWT + Host header"| EP
    EP --> H1
    EP --> H2
    EP --> Hn

    H1 -.CloudEvents.-> XIngest
    H2 -.CloudEvents.-> XIngest
    Hn -.CloudEvents.-> XIngest
    XIngest --> Redpanda --> XConsumer
    XConsumer --> Stripe
    XConsumer --> S3

    Health -->|scale k8s Deployments| H1
    Gateway --> Stripe
    Gateway --> Redis
    Gateway --> Resend
    Gateway --> PostHog
    Gateway --> Sentry
    XIngest --> Redis

    Gateway --> AlloyDB
    Admin --> AlloyDB
    Health --> AlloyDB
    Dashboard --> Supabase
    AlloyDB -->|DB triggers| HR
    HR -->|provisions| Honcho
ServicePortStackPurpose
Groudon Gateway8000FastAPIPublic API passthrough, dashboard backend, Stripe/Honcho webhooks, billing pause/resume
Admin8001FastAPIInternal-only tenant + user CRUD, debugging endpoints
Health9090FastAPI loopsRepair (3 min), upgrades (5 min), hygiene (30 min), placeholder pool, Prometheus metrics
Dashboard3000Next.js 15 / BunTenant self-service: status, metrics, keys, billing, members
Xatu—FastAPI + AIOKafkaCloudEvents ingestion → Redpanda → Stripe usage + S3 archive

Gateway, Admin, and Health share the groudon/ core: SQLAlchemy models (models.py), database access (crud.py, ~1.6k lines), instance credentials (honcho.py), cluster and namespace resolution (k8s_ops.py), URL resolution (routing.py), per-tenant database provisioning (tenant_database.py), and the Stripe, Redis, and PostHog clients.

Instance state machine

tenants.instance_state is what the gateway checks on every request and what the health loops act on. Runbooks refer to these states constantly.

stateDiagram-v2
    [*] --> none : tenant row created
    none --> pending : instance row + DB created
    pending --> live : health loop proves it serves
    pending --> errored : transient failure
    errored --> pending : health loop retries
    live --> updating : version upgrade
    updating --> live : upgrade confirmed
    updating --> failed : upgrade could not recover
    live --> cold_storage : idle, archived
    cold_storage --> pending : traffic arrives, restoring
    live --> deleted : soft delete
    failed --> [*]
    deleted --> [*]
StateGateway behaviourRecovery
noneno instance yetprovisioning
pending503 with Retry-Afterhealth loop proves LIVE
updatingroutes normallyupgrade completes or fails
liveroutes normally—
cold_storage503, schedules a restoretraffic triggers reprovision
errored503retried automatically by the health loop
failed503manual intervention required
deleted404terminal; row kept as a tombstone

failed and errored differ deliberately. errored is a transient failure during creation and is retried. failed is a terminal one, an upgrade that could not recover or an IntegrityError during creation, and is never retried automatically.

A live row is a latch. It proves the transition to serving happened, not that the instance is serving right now.

cold_storage is hibernation for idle tenants. The Groudon Gateway returns 503 and schedules a restore. New requests will put the tenant back through pending while the kubernetes pods are brought up again.

API request routing

The hot path. A client sends Authorization: Bearer hch-v{major}-{nanoid}; the Groudon Gateway resolves it to a cluster and proxies to that cluster’s edge-proxy with a minted Honcho JWT.

sequenceDiagram
    autonumber
    participant Client as Client / Agent SDK
    participant GW as Groudon Gateway 8000
    participant Cache as TTLCache (300s)<br/>+ Redis pub/sub
    participant DB as AlloyDB (groudon)
    participant Billing as Billing Cache (LRU 100)
    participant EP as Cluster edge-proxy<br/>{app_name}.cluster-{N}.{dns}
    participant H as app-api Deployment

    Client->>GW: HTTPS request + Bearer api_key
    GW->>GW: CORS → Prometheus → rate limit
    GW->>Cache: lookup api_key

    alt cache miss
        GW->>DB: crud.get_api_key(api_key)
        DB-->>GW: {app_name, jwt, tenant_id, honcho_version, cluster_index}
        GW->>Cache: store (300s TTL)
    end

    Note over GW: instance_state gate<br/>pending/cold_storage/errored → 503<br/>deleted → 404

    Note over GW: Path matches paid endpoint?<br/>(message batch / dialectic chat)

    alt paid endpoint
        GW->>Billing: get balance (LRU + 5min refresh)
        alt balance ≤ 0
            GW->>Stripe (external): fresh balance check
            alt still ≤ 0
                GW-->>Client: 402 Payment Required
                GW->>H: scale deriver to 0 (background)<br/>Redis SETNX billing:paused:{app}
            end
        end
    end

    GW->>EP: proxy to http://{app_name}.cluster-{N}.{dns}{path}<br/>Authorization: Bearer {jwt}
    EP->>H: nginx map host → app-api.{namespace}.svc.cluster.local:80
    H-->>EP: response
    EP-->>GW: response
    GW->>GW: record Prometheus metrics<br/>(usage, endpoint, latency)
    GW-->>Client: response

    Note over GW: errors → 502/504/500<br/>+ Sentry if enabled

Key implementation notes

  • API key format: hch-v{major}-{nanoid16} (e.g. hch-v2-abcd1234efgh5678). The version prefix lets the gateway pick a Honcho-compatible JWT format and routing strategy. The hch prefix exists because Kubernetes names cannot start with a digit.
  • No signature: keys are looked up directly in Postgres, then cached in an in-memory TTLCache (300 s). Redis pub/sub broadcasts invalidations across gateway instances so rotated keys take effect immediately.
  • Three things called gateway. The Groudon Gateway (:8000) is our FastAPI service. It proxies to a hostname that resolves to that cluster’s GKE Gateway — the cluster load balancer, not our code. The GKE Gateway forwards to the cluster’s edge-proxy, an nginx whose map turns the hostname into app-api.{namespace}.svc.cluster.local:80.
  • Internal URL: http://{app_name}.cluster-{N}.{INTERNAL_DNS_NAME}/. The hostname matches *.cluster-{N}.api.prod.internal. Groudon only needs to know the cluster; the edge-proxy map does the last hop.
  • Two hops, one reason: a single GKE Gateway per cluster is cheaper than an HTTPRoute per tenant, and 500 tenants per cluster would otherwise mean 500 routes. The cost is that a new tenant is not reachable until the edge-proxy ConfigMap is re-rendered.
  • JWT: HS256, signed with the instance’s auth_jwt_secret. Claim names are intentionally short (t, exp, ad, ap, us, se, co) to minimize header size. For the gateway-issued passthrough JWT, ad=true (admin scope).
  • Billing pause is idempotent: SETNX billing:paused:{app_name} with a 30-day TTL ensures only the first concurrent caller pauses the derivers; subsequent requests during a burst skip the pause call. The pause itself scales the app-deriver Deployment to zero. Resume is triggered by the Stripe billing_alert.triggered webhook after auto top-up clears.

Tenant provisioning

Provisioning is split across two trigger points. The dashboard’s /tenants/initialize call creates the org and Stripe customer immediately; the instance is only created once Stripe confirms a payment method via the setup_intent.succeeded webhook.

Groudon does not create Kubernetes resources. It writes rows, and DB triggers hand off to the GitOps pipeline, which renders Helm values and lets ArgoCD apply them. See the GitOps pipeline for that half.

sequenceDiagram
    autonumber
    participant User
    participant Dash as Dashboard (Next.js)
    participant GW as Groudon Gateway
    participant DB as AlloyDB (groudon)
    participant Stripe
    participant TDB as AlloyDB (tenant DB)
    participant Pipe as GitOps pipeline
    participant Health as Health service
    participant H as app-api

    User->>Dash: Create organization
    Dash->>GW: POST /dashboard/v1/tenants/initialize
    GW->>DB: crud.create_tenant() + admin user mapping
    GW->>Stripe: create_customer(tenant_id)
    Stripe-->>GW: customer_id
    GW-->>Dash: tenant created (no instance yet)

    User->>Dash: Add payment method
    Dash->>Stripe: SetupIntent confirmation
    Stripe-->>GW: webhook: setup_intent.succeeded

    alt placeholder available
        GW->>DB: claim a placeholder tenant row
        Note over GW,DB: workload already exists; pods do not restart
        Note over Pipe: a render may still run (values changed) but it is a no-op for pods
    else cold provision
        GW->>TDB: select AlloyDB cluster, create database
        GW->>DB: crud.create_honcho_instance()
        Note over DB: INSERT allocates a cluster/batch slot
        DB->>Pipe: trigger → groudon-health render → argo-production
        Pipe->>Pipe: ArgoCD applies namespace + api + deriver
    end

    GW->>DB: api_key = hch-v{major}-{nanoid}
    GW-->>Dash: tenant created, instance pending

    loop health loop, 3 min
        Health->>DB: read pending instances
        Health->>H: HTTP probe whether it serves
        Health->>DB: flip pending → live, or → errored
    end

Key implementation notes

  • Placeholder pool: the health service maintains NEXT_TENANT_POOL_SIZE un-owned instances in a placeholder=true state. When a real tenant arrives, Groudon flips tenant_id onto that row instead of waiting for a new workload to be rendered and scheduled. The pool is replenished asynchronously. Claiming causes zero pod disruption because the charts do not reference tenant_id: a render may still fire (the values file now lists a different tenant in that slot) but the pods do not restart. The row can still sit in pending until the health loop proves it serves.
  • Slot allocation happens in the database. Inserting a honcho_instances row fires trigger_allocate_tenants(), which assigns the tenant the lowest free (cluster_index, batch_index) slot. Nothing in application code picks a cluster.
  • api and deriver split: two Deployments per namespace. The passthrough only routes to app-api; the billing pause path scales app-deriver. Kubernetes owns replica counts, so Groudon scales Deployments rather than tracking individual workloads.
  • Version pinning: each instance records its honcho_version, which selects both the image and the version-scoped secret path in AWS Secrets Manager. Upgrades are driven from health/ and land as a rendered image change; Honcho announces readiness via a signed webhook back to the gateway.
  • Webhook security: instance-to-gateway webhooks are HMAC-SHA256-signed with each instance’s unique webhook_secret. Replay protection is via timestamp comparison.
  • The machines table is vestigial. It is Fly-era inventory, unused because Kubernetes owns replicas, and should be dropped once nothing depends on it.

Billing and the Xatu pipeline

For Honcho >= v3.0.0, billing is event-driven: each instance emits CloudEvents into Xatu, which dedupes, partitions, and forwards usage to Stripe while archiving the raw stream to S3 for SOC 2 and reconciliation. Older instances still meter through the gateway inline (legacy fallback in passthrough.py).

sequenceDiagram
    autonumber
    participant H as Honcho Instance
    participant XI as Xatu Ingestion
    participant Redis as Redis dedup 48h
    participant RP as Redpanda honcho-events
    participant XC as Xatu Consumer
    participant Stripe
    participant S3
    participant GW as Gateway

    loop per billable operation
        H->>XI: POST /v1/events<br/>X-Signature HMAC<br/>source /honcho/{app_name}/{category}
    end

    XI->>XI: verify HMAC
    XI->>Redis: batch dedup check (event_ids, 48h TTL)
    Redis-->>XI: new vs seen
    XI->>RP: publish key=tenant_id (partition by tenant)
    XI-->>H: 202 Accepted

    Note over RP,XC: consumer_group=honcho-telemetry<br/>manual commit, batches up to 1000

    XC->>RP: poll honcho-events
    RP-->>XC: batch
    XC->>XC: enrich (tenant_id, namespace)

    par to Stripe
        XC->>Stripe: report usage records
    and to S3
        XC->>S3: append Parquet partition<br/>state snapshot - write - verify - commit
    end

    alt processing failure
        XC->>RP: publish to honcho-events-dlq
    end

    Note over Stripe,GW: balance hits threshold

    Stripe-->>GW: webhook billing_alert.triggered
    GW->>Stripe: process_auto_topup_with_balance()

    alt balance > 0 after top-up
        GW->>H: scale derivers back up
        GW->>Redis: DEL billing:paused:{app_name}
    else still 0
        Note over GW: stays paused (402 on next paid call)
    end

Key implementation notes

  • Dedup is efficiency-only. Stripe and the dashboards dedupe by event id downstream, so a Redis outage costs duplicate work rather than double billing. Xatu fails open: if Redis is unreachable, check_duplicates returns an empty set and everything is treated as new.
  • Dedup window: 48-hour TTL on Redis event-ID keys. Events outside the window are treated as new.
  • Partition key: tenant_id. This guarantees per-tenant ordering through Redpanda even with multiple consumer replicas, which matters for the S3 compaction phases (snapshot → write → verify → commit → done).
  • DLQ: honcho-events-dlq. Consumer failures don’t block the main topic; failed events surface to operations dashboards for reconciliation.
  • Reconciliation: S3 partitions carry a state.json describing compaction phase. This is the source of truth if Stripe and Redpanda drift apart, for example during a Stripe outage.

Data model

Groudon’s control-plane state lives in a single AlloyDB schema (default: groudon).

ModelKey fields
UserSupabase auth UUID, email
Tenantid, name, stripe_customer_id, owner_id, instance_state, placeholder
HonchoInstancetenant_id, app_name, honcho_version, region
TenantDatabaseid, AlloyDB cluster, secret_name, status, type
TenantBatchAllocationtenant_id, cluster_index, batch_index, is_allocated
ApiKeytenant_id, prefix (hch-v{N}-), key_id, jwt, scopes
Webhooktenant_id, url, secret, events
HonchoVersionsemver, image_label, deprecated_at
ArgoWebhookQueuename (helm_render / cluster_manager), pending, state
MaintenanceJobscheduled repair, upgrade, and hygiene job bookkeeping
FreeAPIUsagefree-tier metering
MachineFly-era, unused

TenantBatchAllocation is where a tenant’s placement lives, and it is the join the GitOps renderer reads to decide which values file a tenant lands in. cluster_index is stable across an instance’s life, so a tenant returns to the same cluster after a rebuild.

Per-tenant Honcho data lives in each tenant’s own database on an AlloyDB cluster, not in Groudon’s schema.

Cross-cutting concerns

  • Caching: API keys cached in-process (TTLCache 300s) with Redis pub/sub invalidation. Billing balance cached LRU (100 tenants) with a 5-minute background refresh.
  • Rate limiting: gateway-level via SlowAPI middleware. Limits keyed on API key for passthrough, on Supabase user for dashboard routes.
  • Observability: Prometheus on :8000/metrics (gateway) and :9090/metrics (health). Key gauges: HONCHO_API_USAGE, HONCHO_API_LATENCY, BILLING_PAUSED. Sentry for errors. PostHog for product analytics.
  • Auth: Supabase JWT (HS256, with JWKS rotation) for dashboard routes; API key prefix routing for passthrough. MFA enforced on dashboard.
  • Redis is Memorystore for Redis Cluster, three shards reached over Private Service Connect. Clients must be cluster-aware; a non-cluster client against it produces MOVED errors rather than a clean failure.

See Also

Graph View

Backlinks