Where to look

Something is wrong and you do not yet know where to look. Start here, then follow the runbook it points at.

Consoles

Grafana — Authenticates with Google SSO. No username/password needed. prod · staging
Argo CD — Username is admin, password in Secret Manager on the matching GCP project (plastic-labs-prod / plastic-labs-staging). prod · staging

Debugging tools

kubectx / kubens — allow you to easily switch between clusters and namespaces. Install fzf so both tools let you easily fuzzy search the list of clusters and namespaces.
k9s — TUI for pods, logs, and events on the current context.

Map

Is a tenant’s workload synced and healthy — ArgoCD
System-wide request rates, latency, error codes — Grafana - see Groudon Metrics dashboard.
Database saturation, connections, restarts — AlloyDB in GCP Cloud Console for either the orchestrator or a tenant’s database.
Logs from a pod — kubectl logs
Errors and stack traces — Sentry
Whether a cluster got built — Cloud Build + cluster-manager logs
Whether the renderer is running — groudon-health on the operations cluster
Redis pressure — Memorystore

Getting onto a cluster

Contexts are named gke_plastic-labs-prod_us-east4_cluster-N and gke_plastic-labs-prod_us-east4_operations. If kubectl hangs, the cluster is likely fine but your kubeconfig might be stale. Run the following command to get the list of clusters and fetch their credentials:

gcloud container clusters list --project=plastic-labs-prod --format="value(name,status)"
gcloud container clusters get-credentials cluster-38 --region=us-east4 --project=plastic-labs-prod # replace with the cluster name

After that, kubectx / kubens (with fzf) instead of --context / -n on every command, and k9s for the current context.

A tenant lives in one namespace. The name carries cluster, batch, and instance:

cluster-38-batch-23-hch4p8a9ct3nibpl15rk207c

Finding the tenant from an alert is also covered in tenant-debugging.

Why a tenant is not reachable

Routing is two hops. The Groudon Gateway resolves the tenant to a cluster; that cluster’s edge-proxy maps the hostname to a namespace. A newly provisioned tenant is unreachable until the proxy’s ConfigMap re-renders.

Compare desired against live:

# desired
git -C ~/plastic/kubernetes/argo-production show \
  origin/main:clusters/cluster-44/networking-values.yaml | grep -c "app_name:"
 
# live
kubectl --context gke_plastic-labs-prod_us-east4_cluster-44 -n networking \
  get cm edge-proxy-nginx-conf -o jsonpath='{.data.nginx\.conf}' \
| grep -c 'api.prod.internal" "app-api'

A gap means the ConfigMap is stale, which means the networking Application is not syncing. This is fleet-wide when it happens: one unappliable resource in sync wave 0 stops the wave-1 ConfigMap on all 46 clusters while every Application still reports Healthy. See GitOps pipeline.

Request rates, latency, error codes

Use the GKE Grafana, not Fly. URLs and SSO are in Consoles above.

Mimir itself, from a laptop:

kubectl --context gke_plastic-labs-prod_us-east4_operations -n metrics \
  port-forward svc/mimir 8080:8080
 
curl -sG http://localhost:8080/prometheus/api/v1/query \
  -H "X-Scope-OrgID: anonymous" \
  --data-urlencode 'query=sum(rate(telemetry_events_emitted_total[10m]))'

The X-Scope-OrgID header is required; without it you get an empty result rather than an error. In-cluster the address is mimir.metrics.prod.internal:8080. Topology is in Infra.

Metrics worth knowing: honcho_endpoint_counts_total (per-tenant request counts by resource and status_code), telemetry_events_emitted_total (Xatu event rate), kube_pod_status_phase (what the Pending alert fires on).

Database saturation, connections, restarts

This is Cloud Console, not a SQL prompt. There are ~20 AlloyDB HA clusters. Pick the cluster, then System insights / Connections. A tenant’s cluster is on its TenantDatabase row, not in the namespace name.

Do not confuse this with Groudon’s orchestrator database. Orchestrator saturation shows up as Groudon itself wedging; tenant saturation shows up as one Honcho instance failing to connect. The failure mode is AlloyDB connection saturation.

Connecting to the orchestrator DB

Groudon’s own state (tenants, placements, versions) is in AlloyDB cluster prod-orchestrator-alloy-db, database groudon, user postgres. The password is in Secret Manager as prod-orchestrator-alloy-db-alloydb-password in plastic-labs-prod. The primary has a public IP; take it from the console or:

gcloud alloydb instances describe prod-orchestrator-alloy-db-primary \
  --cluster prod-orchestrator-alloy-db --region us-east4 --project plastic-labs-prod \
  --format='value(publicIpAddress)'

Groudon scripts that take --prod (scale_tenant_deployment, force_update_honcho) read the connection from the repo .env:

PROD_DB_URI=postgresql+psycopg://postgres:<url-encoded password>@<ip>:5432/groudon
GCP_PROJECT=plastic-labs-prod
DB_POOL_SIZE=10
DB_MAX_OVERFLOW=5

The password has to be percent-encoded (@ becomes %40, ( becomes %28, ! becomes %21); the scripts do not re-encode it, and a raw @ makes the URI parse the wrong host. DB_POOL_SIZE and DB_MAX_OVERFLOW are read at import, so their absence fails as a KeyError before anything connects. Cluster access uses application default credentials: gcloud auth application-default login.

Logs from a pod

If the pod is still there, including crashlooping:

kubectl --context $C -n $NS logs <pod> --tail=80
kubectl --context $C -n $NS logs <pod> --previous --tail=40

--previous is the container that died. Without it you get the current retry, which has not failed yet. Which exit reason means what is Tenant pod crashlooping.

If the pod is already gone, kubectl logs has nothing. Vector ships from every node to the vector-aggregator in log-ingest on the operations cluster, which sinks to Pub/Sub. That is the historical path. In-cluster: vector-aggregator.log-ingest.prod.internal.

Probes are localhost-only. A pod can be 1/1 Running and still be dead to the network. See Node loses egress.

Did a cluster get built

gcloud builds list --project=plastic-labs-prod --limit=5 \
  --format="value[separator='  '](id,status,createTime)"
 
gcloud builds log <id> --project=plastic-labs-prod 2>&1 \
| sed 's/\x1b\[[0-9;]*m//g' | grep -nE "^Step #2: .?.?Error|resourceInUse|already being used"

The log is 800 lines and mostly image pulls, so grep or you will scroll. Terraform errors are all in Step #2.

The service’s own reasoning is in Cloud Run logs, and it is the fastest way to see what it thinks it should do:

gcloud logging read \
  'resource.labels.service_name="cluster-manager" AND jsonPayload.message:"clusters"' \
  --project=plastic-labs-prod --limit=6 --freshness=1h \
  --format="value(timestamp,jsonPayload.message)"

Is the renderer running

Rendering lives inside groudon-health on the operations cluster, not on Cloud Run. The render job ticks every 15s when argo_webhook_queue.helm_render is pending, and also on a 10-minute staleness floor, so a quiet gap between commits to argo-production is normal. The leftover Cloud Run helm-render service is not called.

kubectl --context gke_plastic-labs-prod_us-east4_operations -n groudon \
  logs deploy/groudon-health --tail=80

webhook-listen on Cloud Run only drives cluster-manager. Its logs say nothing about tenant renders.

Which image is actually running

for s in cluster-manager webhook-listen; do
  echo -n "$s: "
  gcloud run services describe $s --project=plastic-labs-prod --region=us-east4 \
    --format="value(spec.template.spec.containers[0].image)" | sed 's/.*://'
done

Compare against git rev-parse origin/stable for prod, origin/master for staging.

Groudon, Xatu, and the renderer run on GKE, so their version is the image tag on the operations cluster:

kubectl --context gke_plastic-labs-prod_us-east4_operations -n groudon \
  get deploy -o custom-columns='NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image'

What is occupying an IP range

gcloud compute networks subnets list --project=plastic-labs-prod --regions=us-east4 \
  --format="value[separator='  '](name,ipCidrRange)"
 
gcloud compute addresses list --project=plastic-labs-prod \
  --format="value[separator='  '](name,address,purpose,status,subnetwork.basename())"

If a range appears in neither but a subnet create still fails with “overlaps with an active peer network”, it belongs to a Google producer project and is only visible as a peering route:

gcloud compute networks peerings list-routes servicenetworking-googleapis-com \
  --network=prod-plastic-labs-network --region=us-east4 --direction=INCOMING \
  --project=plastic-labs-prod

That is how the orphaned 10.174.x ranges were found. See Infra for the full plan.

Signals that lie

The ones that cost the most time, because they look definitive.

kubectl timing out does not mean the cluster is down. A recreated cluster has a new endpoint and CA; a stale kubeconfig gives Unable to connect to the server: context deadline exceeded. Cross-check with gcloud container clusters list, which will happily show it RUNNING with nodes. Fix by re-fetching credentials.

Pod health says nothing about dependencies. Xatu’s /health returns healthy unconditionally and its dedup catches every exception, so a dead Redis leaves pods 1/1 Running with dedup silently disabled. The August node-egress incident was the same shape: ~28 pods Running and passing probes while every outbound call timed out, because probes are localhost-only.

Plastic Labs Internal Wiki

Explorer

Recently Updated

Company

Operating Model — Decisions Log

Operating Model

Security & Compliance

ideal-reasoned-representations