Groudon GitOps Pipeline

Read this before changing anything in argo-source, argo-production, or gke-cluster-manager, and before debugging why a tenant is not serving.

What this is

GitOps means git is the source of truth. Nobody types kubectl apply to change production. We commit YAML to a repo, and a controller makes the clusters match what is in git.

The pieces:

A cluster is a pool of kubernetes resources. Ours are named cluster-0 through cluster-45, plus one operations cluster that runs shared services.
A namespace is one tenant’s sandbox inside a cluster. Tenants on the same cluster cannot see each other’s workloads.
A Helm chart is a directory of templates: “this is what a tenant (or a cluster) looks like.”
Helm values are the filled-in numbers for a specific tenant or batch — image tag, secrets, whether it should be running.
ArgoCD is the controller. It watches those git repos and applies the difference to the clusters.

There are two git repos you will touch:

argo-source — the charts (templates). Edited by hand.
argo-production / argo-staging — the values (filled-in numbers) plus the ArgoCD config that says which chart goes where. Most of this is machine-written; some of it is edited by hand.

gke-cluster-manager is not a repo ArgoCD watches. It is a Cloud Run service that creates new GKE clusters when we run out of room.

Two pipelines, one control plane

There are two independent loops, both driven by writes to Groudon’s Postgres. They are easy to confuse because both get called “the GitOps pipeline” in conversation.

flowchart TB
    subgraph DB["Groudon Postgres (AlloyDB)"]
        HI["honcho_instances"]
        TBA["tenant_batch_allocations"]
        Q["argo_webhook_queue"]
    end

    subgraph Poll["webhook-listen (Cloud Run, minScale 1)<br/>drives cluster_manager only"]
        W["polls argo_webhook_queue"]
    end

    subgraph Tenant["Tenant updates"]
        HR["render job in groudon-health<br/>DB → Helm values"]
        AP["argo-production<br/>values + Application defs"]
        AS["argo-source<br/>Helm charts"]
        ACD["ArgoCD (operations cluster)"]
        T["cluster-0 … cluster-45<br/>one namespace per tenant"]
    end

    subgraph Provision["Cluster provisioning"]
        CM["cluster-manager (Cloud Run)<br/>desired vs existing"]
        CB["Cloud Build<br/>terraform apply"]
        GKE["new GKE Autopilot cluster<br/>+ subnet + gateway IP + DNS"]
    end

    HI -->|AFTER INSERT OR UPDATE| Q
    TBA -->|AFTER UPDATE OF cluster_index| Q
    Q -->|polled every 15s| HR
    Q -->|cluster_manager slot only| W
    W -->|GET /run| CM

    HR -->|commit| AP
    AP --> ACD
    AS --> ACD
    ACD -->|sync| T

    CM --> CB --> GKE
    GKE -->|registered with ArgoCD| ACD

Tenant updates turns database rows into running tenants. A maintenance job inside groudon-health reads allocated tenants, writes YAML into argo-production, and ArgoCD deploys it. This runs tens to hundreds of times a day.

Cluster provisioning adds a new GKE cluster when a tenant is allocated onto one that does not exist yet. In conversation this is also called “adding a new cluster.”

What triggers a render

A render is the renderer reading allocated tenants from Postgres and committing YAML files to argo-production (or argo-staging).

It is primarily event-driven: writes to the groudon schema in our AlloyDB orchestrator cluster set argo_webhook_queue.helm_render pending, via a database trigger on honcho_instances. There is also a 10-minute staleness floor, so a render still happens periodically when nothing has changed — it just finds no diff and commits nothing.

A quiet gap between commits to argo-production is therefore normal, not a stall. Rendering is deterministic, so an unchanged fleet produces an identical tree.

The renderer: database to Helm values

Rendering runs inside the groudon-health deployment on the operations cluster (groudon/helm_render/, driven by health/maintenance.py), not on Cloud Run. The render maintenance job ticks every 15s and renders when the helm_render queue row is pending, or when nothing has rendered for RENDER_STALE_AFTER_SECONDS (600s) — so a render happens within 10 minutes regardless.

Historical: this used to be a Cloud Run service (helm-pre-render) fired by webhook-listen. The service still exists but nothing calls it — the poller’s queue slot for it reads disabled in both environments. It was moved in-process so rendering is deterministic and shares one set of models with Groudon.

It reads from the groudon schema and writes YAML files to argo-production or argo-staging, depending on the environment.

It looks at allocated tenants that are not deleted. pending and updating are both written out as live, because the chart only cares whether the tenant should be running.

Tenants are grouped into batches. A batch is a group of tenants that share one ArgoCD Application, so they get updated together. Production has 50 batches per cluster and 10 tenants per batch. Staging has 2 batches per cluster and 5 tenants per batch.

The files that land in argo-production / argo-staging:

clusters/cluster-0/batch-0-values.yaml … batch-49-values.yaml — one file per batch.
clusters/cluster-0/networking-values.yaml — every tenant on that cluster, used by the edge proxy to know where to send traffic. One file per cluster.

Under the hood the query joins HonchoInstance, TenantBatchAllocation, Tenant, TenantDatabase, and HonchoVersions, filtered on is_allocated = true.

The ArgoCD repos

argo-source holds the Helm charts — the templates.

Chart	Applied to	What it is
`base/`	every tenant namespace	The tenant itself: its namespace, secrets, API, and deriver
`base_networking/`	every tenant cluster	How traffic gets in: the router, the edge proxy and its tenant map, plus compute class and AWS secrets plumbing
`observability-agent/`	every cluster	Log and metrics agents on each machine
`central-metrics/`	operations cluster only	Grafana and Mimir
`central-observability/`	operations cluster only	The log aggregator

argo-production and argo-staging hold the generated values plus the ArgoCD config that says which chart goes where:

app-of-apps.yaml              the root Application (tenants, Groudon, Xatu, metrics)
app/Applicationset.yaml       every ApplicationSet and singleton Application
clusters/cluster-N/           generated values, one dir per cluster
groudon/                      gateway, health, migrate job, secrets
xatu/                         ingestion, consumer, secrets

Only clusters/ is machine-written. Everything else is edited by hand.

ArgoCD

ArgoCD runs on the operations cluster — one Helm release per environment, applied by hand from infra-operations. It does not manage itself. It watches the two git repos and makes every cluster match what is in git: apply what is missing, delete what should not be there, overwrite anything someone changed by hand.

The objects it uses:

An Application is ArgoCD’s record of “this path in git should look like this on that cluster.”
An ApplicationSet is a factory. It looks at files matching a glob and creates one Application per file. That is why dropping batch-17-values.yaml into git is enough — nobody creates the Application by hand.
app-of-apps is the root Application. It points at app/Applicationset.yaml, and from there ArgoCD manages the tenant Applications, Groudon, Xatu, and the metrics stack — not the ArgoCD controller itself.
prune + selfHeal on that root means: if it is not in git, delete it; if someone changed the cluster by hand, overwrite it.

What that tree looks like:

app-of-apps  (Application, prune + selfHeal)
  └─ app/Applicationset.yaml
       ├─ ApplicationSet tenant-batches-production
       │    watches clusters/*/batch-*-values.yaml
       │    → one Application per file, using chart base/
       ├─ ApplicationSet cluster-networking-production
       │    watches clusters/*/networking-values.yaml
       │    → one Application per file, using charts
       │      base_networking/ + observability-agent/
       ├─ Application groudon-production                 → operations
       ├─ Application xatu-production                    → operations
       ├─ Application central-metrics-production         → operations
       ├─ Application central-observability-production   → operations
       └─ Application observability-agent-operations     → operations

The two ApplicationSets are the ones that scale with the fleet. The five Applications at the bottom are singletons on the operations cluster.

Sync waves

ArgoCD applies resources in numbered waves: 0 first, then 1, then 2, and so on. If a resource in one wave fails to apply, every later wave is skipped.

Wave	Tenant chart (`base/`)	Cluster networking (`base_networking/`)
0	`namespace`	(nothing — unless someone adds a resource without a wave annotation)
1	`honcho-secrets`	`edge-proxy-configmap` (the tenant map)
2	`honcho-api`	`edge-proxy`, `router`
3	`honcho-deriver`

The networking chart is the one to remember. The ConfigMap that tells the edge proxy about tenants is wave 1. Any resource you add without a sync-wave annotation defaults to wave 0 — before that ConfigMap. If GKE refuses the resource, the tenant map never updates, on every cluster, while ArgoCD still reports every Application as Healthy.

gke-cluster-manager: adding clusters

Cluster provisioning is a separate Cloud Run service. It does not go through ArgoCD. When it runs, it:

Figures out how many clusters we should have
Diffs that against what actually exists in GKE
Generates a terraform root, uploads it to GCS, and kicks off Cloud Build to create the new cluster(s).

How many clusters we want

The desired count comes from cluster/cluster_db.py:

PLACEHOLDER_CLUSTER = 0 if IS_STAGING else 1
return int(max_cluster_index) + OFF_BY_ONE_COMPENSATION + PLACEHOLDER_CLUSTER

Production keeps one spare cluster ahead of demand. Staging does not, which is why staging sits right against its ceiling.

OVERRIDE_NUMBER_OF_CLUSTERS skips the query entirely. Use it to freeze fleet growth during an incident.

Which clusters are missing

This is a live diff against gcloud container clusters list, not terraform state:

existing_indices = _parse_existing_cluster_indices(
    project, name_prefix
)  # gcloud container clusters list
missing_indices = sorted(set(range(desired_total)) - existing_indices)

If you delete a cluster by hand, the next run treats it as new. Those indices are written to scripts/new_clusters.txt so the post-apply steps know which clusters to register.

What Cloud Build does

Three steps:

Fetch the generated tarball
terraform init and terraform apply -auto-approve
Run register-new-clusters.sh and apply-workload-sa.sh against only the clusters in new_clusters.txt

Registration is kubectl apply of a secret with a fixed name. Recreating a cluster overwrites the old endpoint and CA instead of leaving a duplicate.

How cluster IPs are assigned

Each cluster gets a /21 of the form 10.(octet).0.0/21, starting at octet 128. Octets listed in reserved_octets are skipped:

cluster_octet = {
  for i in range(var.max_clusters) : i =>
  var.cluster_base_octet + i + length([
    for o in var.reserved_octets : o if o <= var.cluster_base_octet + i
  ])
}

reserved_octets = [174] exists because three orphaned /24s from deleted AlloyDB clusters still live in Google’s producer project. They do not show up in our VPC, but creating a subnet there fails with “overlaps with an active peer network”. That is why cluster-46 sits on 10.175 rather than 10.174.

Each cluster also gets DNS:

cluster-N.prod.internal for the control plane
*.cluster-N.api.prod.internal for the gateway IP

Both records live in the hand-created prod-internal Cloud DNS zone.

Traps

These have cost real time, and each one looks like something else.

A failed wave 0 freezes tenant routing everywhere. Example: a kube-system PodDisruptionBudget added without a sync-wave annotation. GKE Autopilot’s Warden owns that namespace and denies the resource. Wave 0 fails, wave 1 never applies, and the edge-proxy tenant map stops updating on all 46 clusters. Every Application still reports Healthy.

OutOfSync is not a reliable signal. The renderer writes files in nondeterministic order, so Applications flap constantly. Real drift hides in that noise.

Autopilot rewrites resource requests at admission. Under the Default compute class, GKE requires 6.5 GiB of memory per vCPU. A container that asks for 500Mi of memory and 63m CPU gets rewritten to 76m CPU.

Promotion to production is manual. gke-cluster-manager deploys staging from master and production from stable. A fix merged to master hits staging immediately. Production only gets it when someone promotes stable.

Cloud Build submission is not Cloud Build success. The /run endpoint returns as soon as the build is submitted. A 200 tells you nothing about whether terraform actually applied.

Terraform plans the whole root. There is no per-cluster apply. A change that renumbers subnets tries to replace every subnet in one operation. Tearing down one cluster removes one blocker; it does not shrink the blast radius.

Plastic Labs Internal Wiki

Explorer

Recently Updated

Company

Operating Model — Decisions Log

Operating Model

Security & Compliance

ideal-reasoned-representations