Plastic Labs Internal Wiki

Recently Updated

Tenant debugging

5 min read

Tenant debugging

How to find a tenant’s pods, work out why one is restarting, and decide whether it is that tenant’s problem or something wider.

Most alerts land here first. The alert names a cluster and a namespace and nothing else.

Related: node-egress-failure · alloydb-connection-saturation · honcho-debugging

Finding the tenant

A tenant lives in one namespace on one cluster. The namespace name carries everything you need:

cluster-38-batch-23-hch4p8a9ct3nibpl15rk207c
└─cluster─┘└─batch─┘└──────app_name─────────┘

Alerts give you cluster and namespace directly, in resource.labels. Set the context from the cluster number (or use kubectx):

C=gke_plastic-labs-prod_us-east4_cluster-38
NS=cluster-38-batch-23-hch4p8a9ct3nibpl15rk207c
kubectl --context $C -n $NS get pods

Going the other way, from an app_name with no cluster, search the generated values in argo-production:

cd argo-production
git grep -l "hch4p8a9ct3nibpl15rk207c" origin/main -- 'clusters/*/networking-values.yaml'

That returns clusters/cluster-38/networking-values.yaml, which gives you the cluster. The same file holds the tenant’s instance_state, version, image_label, and namespace.

Or you can search for the tenant id within the tenant batch allocations table in the Groudon schema in our AlloyDB Orchestrator DB. Connecting to it is covered in Where to look.

select * from groudon.tenant_batch_allocations where tenant_id = 'TENANT ID';

This will return the cluster index and batch index for the tenant.

Each tenant runs two deployments. app-api serves requests. app-deriver processes the queue. A tenant in cold storage shows both scaled to 0, which is normal. A live tenant with either deployment at 0 is not normal: both Argo (from instance_state in the rendered values) and Groudon (k8s_ops, on cold-storage restore, billing pause, and errored repair) write spec.replicas, and an Argo sync that started before a Groudon scale-up can apply a stale 0 on top of it.

Find out who scaled it and when from the GKE audit log:

gcloud logging read --project=plastic-labs-prod --freshness=6h \
  'logName="projects/plastic-labs-prod/logs/cloudaudit.googleapis.com%2Factivity"
   AND resource.labels.cluster_name="cluster-38"
   AND protoPayload.resourceName:"namespaces/'$NS'/deployments/"
   AND protoPayload.request.spec.replicas:*' \
  --format='value(timestamp,protoPayload.resourceName.basename(),protoPayload.request.spec.replicas,protoPayload.requestMetadata.callerSuppliedUserAgent)'

A user agent of argocd-application-controller is Argo; OpenAPI-Generator is Groudon’s Python client. To put a deployment back, use the script rather than kubectl scale, so the change is logged with the same user agent as Groudon’s own writes:

cd groudon
uv run python -m groudon.scripts.scale_tenant_deployment --prod --tenant-id <id> --deployment app-deriver --replicas 1

It resolves the namespace from the orchestrator DB and asks for confirmation before patching.

Start with the exit reason

The status column tells you which layer failed. Read lastState before reading anything else, because the current container is a fresh retry with no history.

kubectl --context $C -n $NS get pod <pod> \
  -o jsonpath='{range .status.initContainerStatuses[*]}init/{.name} restarts={.restartCount} {.lastState.terminated.reason} exit={.lastState.terminated.exitCode}{"\n"}{end}{range .status.containerStatuses[*]}{.name} restarts={.restartCount} {.lastState.terminated.reason} exit={.lastState.terminated.exitCode}{"\n"}{end}'

Then read the logs of the container that died:

kubectl --context $C -n $NS logs <pod> --previous --tail=40
kubectl --context $C -n $NS logs <pod> -c provision-db --previous --tail=40   # for an init container

--previous is required. Without it you get the logs of a container that has not failed yet.

Example Failure Reasons

exit 1 at startup, with a network timeout

ConnectTimeout: HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', ...)

The pod cannot reach the internet. Check whether other broken pods share its node before touching the tenant. See node-egress-failure.

exit 0, reason Completed, on a healthy-looking process

The container was killed by a failed liveness probe, then shut down gracefully. Look at the probe timeout and the pod events:

kubectl --context $C -n $NS describe pod <pod> | grep -A5 -E "Liveness|Readiness|Events"

exit 137, OOMKilled, on the main container

Real memory pressure. Compare current usage against the limit, and check whether requests are set far below actual usage, which makes Autopilot pack too many pods onto a node.

kubectl --context $C -n $NS top pod
kubectl --context $C -n $NS get deploy <name> -o jsonpath='{.spec.template.spec.containers[0].resources}'

Database connection errors

(psycopg.errors.ProtocolViolation) server login has been failing, cached error:
remaining connection slots are reserved ...

Not a tenant problem. Their AlloyDB cluster is out of connection slots. See alloydb-connection-saturation.

Is it wider than this tenant?

Answer this before spending time on a single namespace.

# Other broken pods in the same cluster?
kubectl --context $C get pods -A --no-headers | awk '$1 ~ /batch/ && $4 != "Running"'
 
# Do the broken pods share a node?
kubectl --context $C -n $NS get pod <pod> -o jsonpath='{.spec.nodeName}'
kubectl --context $C get pods -A --field-selector spec.nodeName=<node> --no-headers | awk '{print $4}' | sort | uniq -c
 
# Other clusters affected?
for n in 5 7 20; do # example list of clusters to check
  printf 'cluster-%s: ' $n
  kubectl --context gke_plastic-labs-prod_us-east4_cluster-$n get pods -A --no-headers 2>/dev/null \
    | awk '$1 ~ /batch/ && $4 != "Running"' | wc -l
done

Two unrelated tenants failing in one cluster at the same time is rarely a coincidence. Check the node first.

Checking tenant health beyond the pod

The pod being Running does not mean the tenant is serving. Probes only test localhost.

# Is the instance answering through the gateway?
kubectl --context $C -n $NS logs <api-pod> --since=10m | grep -E '"(GET|POST)' | tail -20
 
# Is it erroring while looking healthy?
kubectl --context $C -n $NS logs <pod> --since=20m | grep -icE "timeout|APIConnection|ConnectError|Traceback"

Graph View

Backlinks