Architectural decisions for Groudon and the managed hosting layer. Newest on top. Format: Architecture Decision Records.
Pin one compute class in production [08.2026]
Context. Autopilot picks a machine shape per pod. Left alone it mixed node types at varying densities, so a 500-tenant cluster used far more nodes than its pod count needed. Fleet-wide: 101 nodes across 15 clusters.
Decision. Pin n4-n2d-highmem-110 on production app-api and app-deriver via a
cloud.google.com/compute-class nodeSelector in the base/ chart. 110 pod slots per
node (n4-highmem-8 or n2d-highmem-8). 101 nodes became 56, roughly $4,800/mo.
Status. Shipped in production. Not applied to staging: ~10 tenants per cluster would waste most of a 110-slot node.
Consequences. One node shape and a large cost cut. The chart now has a
production-only branch. Outside production, Autopilot’s Default class caps memory at
6.5 GiB per vCPU and rewrites requests that exceed it, so the Application never reaches
Synced.
One edge-proxy per cluster, not an HTTPRoute per tenant [07.2026]
Context. Each tenant needs its own hostname in its own namespace. One HTTPRoute per tenant is 500 routes per cluster, ~23,000 fleet-wide. This is very expensive and also very error prone for kubernetes to manage.
Decision. One GKE Gateway per cluster, one wildcard HTTPRoute for
*.cluster-{N}.api.prod.internal, forwarding to a per-cluster nginx edge-proxy. The
proxy maps hostname → app-api.{namespace}.svc.cluster.local from the cluster’s tenant
list. Reload is SIGHUP from a sidecar, not a pod roll: an earlier design keyed a
checksum/config annotation to the ConfigMap and restarted every nginx pod on every
tenant-list change.
Status. Shipped.
Consequences. Route count stays constant as tenants grow. Groudon only needs the
tenant’s cluster, not its namespace. Costs: a new tenant is unreachable until the
ConfigMap re-renders, and that ConfigMap is a fleet-wide chokepoint — if it stops
syncing, no new tenant on any cluster becomes reachable, while every Application stays
Healthy. That failure has happened.
GitOps rather than a Kubernetes operator [05.2026]
Context. Supersedes the operator half of Fly.io → GCP. An operator means owning a reconciliation loop, CRDs, and leader election, and putting cluster mutation on a service that also serves customer traffic.
Decision. No operator. Groudon writes rows. A renderer turns the database into Helm
values and commits them. ArgoCD applies them. Cluster creation is a separate Cloud Run
service (terraform via Cloud Build). Tenants land in a (cluster_index, batch_index)
slot via a database trigger; a batch exists so ArgoCD has a unit of work smaller than a
whole cluster.
Status. Shipped. Mechanics: the GitOps pipeline.
Consequences. Groudon never calls the Kubernetes API to provision, so a control-plane
outage cannot leave workloads half-created. Desired state lives in git and converges
when ArgoCD returns. Provisioning is asynchronous: pending until the health loop
proves it serves. “The row exists” no longer means “it is running.”
Costs: git is now in the onboarding path, so a stalled renderer or failed sync silently
stops new tenants. Debugging spans four repos and two Cloud Run services. A failed sync
wave can leave a fleet-wide resource stale while everything reports Healthy.
Connection-saturation root cause [06.2026]
Context. A ~4.5-hour outage on the shared Postgres: Supavisor client connections spiked
~300 → ~9,000 (EMAXCONN) and the pooler collapsed.
Decision. Confirmed (via pg_cron samplers) the mechanism is connection saturation, not
lock contention. Added a standing pooler-proof monitor and diagnostics. Details:
Connection-saturation runbook and the
deriver polling resilience fix.
Status. Resolved by the move to per-tenant AlloyDB databases, which removed the shared pooler entirely.
Fly.io → GCP (GKE + AlloyDB) [03.2026]
Context. Groudon runs one Fly app per tenant (API + deriver machines). At ~1,500 tenants growing toward 10k, three things break: the shared Supabase Postgres pooler (Supavisor) becomes the bottleneck, Fly’s API isn’t built to orchestrate 10k apps / 40k+ machines, and there’s no way to hibernate idle tenants. The orchestration layer is also locked to Fly, which blocks the cloud-marketplace strategy.
Decision. Migrate Honcho hosting to GCP: GKE for instances (a Groudon Kubernetes operator replaces the Fly orchestration), and AlloyDB for per-tenant database isolation (replacing the shared Supabase schema). Adds auto-hibernation (scale-to-zero + wake).
Status. Shipped. 46 GKE Autopilot clusters, ~22,300 instances, one AlloyDB database per
instance. The operator half was superseded by
GitOps rather than a Kubernetes operator.
Hibernation shipped as the cold_storage instance state.
Consequences. Removed the noisy-neighbor and pooler limits, enabled marketplace portability, and cut idle compute. Cost: a live migration of every Fly tenant, and a new class of failure to learn — Autopilot admission rewriting resource requests, per-cluster IP allocation, and ArgoCD sync ordering.