Plastic Labs Internal Wiki

Recently Updated

Node loses egress (pods stay Running while broken)

6 min read

Node loses egress

A GKE node stops reaching the public internet. Internal traffic keeps working. Most pods on that node stay 1/1 Running and pass every probe while failing every LLM call.

First seen 2026-08-08 on gk3-cluster-6-nap-l8u1qw4k-78f58787-lj6b in cluster-6. The fault ran for about 7 hours before anyone noticed.

Related: Tenant pod crashlooping · alloydb-connection-saturation

Why this is hard to see

Honcho calls load_tiktoken_bpe() at startup, which downloads an encoding file from openaipublic.blob.core.windows.net. Pods created on a broken node crashloop and fire alerts. Pods that were already running before the egress failure were silently degraded. Logs showed that external calls were failing but we were not alerted to the problem.

Three reasons nothing caught it:

  1. Readiness and liveness probes hit localhost. They cannot see egress.
  2. GKE node conditions stayed green. Node-problem-detector does not test egress.
  3. Sentry could not receive the errors, because its ingest endpoint is external and needs the same broken path.

The crashlooping pods later rescheduled onto healthy nodes and the alerts cleared on their own. The node stayed schedulable, so new pods kept landing on it and repeating the cycle.

What ended it was a cordon and a drain. Cordon stopped new pods landing on the node. Drain moved the silently degraded pods off it. Autopilot then deleted the VM and created a replacement, and a rescan of the cluster came back clean. See Fix for the commands.

Symptoms

  • CrashLoopBackOff or “zero available replicas” alerts on two or more unrelated tenants in one cluster, arriving together.
  • The broken pod is minutes old. Its sibling in the same namespace is days old and healthy.
  • A different component fails in each namespace, for example api in one and deriver in another.
  • Crash logs show an outbound call failing, usually tiktoken, ConnectTimeout, or Max retries exceeded.
  • Alerts that resolve without anyone doing anything.

Triage

C=gke_plastic-labs-prod_us-east4_cluster-<N>

1. Check the pod ages. A young broken pod beside an old healthy one points at the environment rather than the tenant.

kubectl --context $C -n <ns> get pods

2. Read the previous container, not the current one. The running container is a fresh retry.

kubectl --context $C -n <ns> get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState}'
kubectl --context $C -n <ns> logs <pod> --previous --tail=30

3. Find out whether the broken pods share a node. This is the step that turns a confusing alert into a located fault.

kubectl --context $C -n <ns> get pod <pod> -o jsonpath='{.spec.nodeName}'
kubectl --context $C get pods -A --field-selector spec.nodeName=<node> --no-headers | awk '{print $4}' | sort | uniq -c

4. Probe egress from a healthy pod on that node. A crashlooping pod has no running container, so it cannot be exec’d.

kubectl --context $C get pods -A -o wide --no-headers \
  | awk -v n=<node> '$8==n && $4=="Running" && $1 ~ /batch/ {print $1, $2; exit}'
 
kubectl --context $C -n <ns> exec <healthy-pod> -- python -c "
import socket, time
def t(label, host, port):
    s=time.time()
    try:
        c=socket.create_connection((host,port), timeout=6); c.close(); print(f'{label:22} OK   {time.time()-s:.2f}s')
    except Exception as e: print(f'{label:22} FAIL {time.time()-s:.2f}s {type(e).__name__}')
print('DNS google ->', socket.gethostbyname('www.google.com'))
t('raw IP (no DNS)', '8.8.8.8', 53)
t('internal kube-api', 'kubernetes.default.svc.cluster.local', 443)
t('litellm (external)', 'tentacruel-litellm-prod.fly.dev', 443)
"

5. Run the same probe from a pod on a different node. A result only means something against a control.

6. Check whether the Running pods are also failing. This exposes the real blast radius.

kubectl --context $C get pods -A --field-selector spec.nodeName=<node> --no-headers \
  | awk '$4=="Running" && $1 ~ /batch/ {print $1, $2}' | head -5 | while read ns pod; do
    printf '%-46s ' "$pod"
    kubectl --context $C -n $ns logs $pod --since=20m 2>/dev/null | grep -icE "timeout|APIConnection|ConnectError"
  done

Rules that keep you honest

Test a raw IP alongside a hostname. DNS failure and routing failure both look like a timeout, and mixing them up sends you down the wrong path.

Generalize the failing call. The hostname in a stack trace is rarely the subject. When google.com fails too, the node is at fault rather than the vendor.

Fleet scan

Download the script: egress_scan.sh

It picks one Running tenant pod per node, probes the LiteLLM host from inside it, and prints OK or FAIL per node. A full sweep of 39 clusters takes about 2 minutes.

chmod +x egress_scan.sh
./egress_scan.sh 6                                              # one cluster
for n in $(seq 0 38); do ./egress_scan.sh $n; done | grep -v '^OK '   # whole fleet

Read stderr as well as stdout. The script exits 2 and says so when it finds no pods at all, which usually means expired credentials rather than a healthy cluster. It warns on stderr when it probes fewer nodes than the cluster has, because a node with no Running tenant pod cannot be checked this way. UNKNOWN on a line means the probe did not run rather than the node being fine.

A scan of 514 nodes on 2026-08-08 found exactly one bad node, which made this an isolated VM fault rather than a VPC or firewall problem.

Fix

# Cordon first. This stops new pods landing in the hole and ends the alert drip.
kubectl --context $C cordon <node>
 
# Drain. This moves the silently degraded pods to healthy nodes.
kubectl --context $C drain <node> --ignore-daemonsets --delete-emptydir-data --grace-period=60
 
# Autopilot reclaims and replaces the VM. Verify:
kubectl --context $C get node <node>          # expect NotFound
./egress_scan.sh <N> | grep -v '^OK '          # expect empty
kubectl --context $C get pods -A | grep -v Running

Deleting only the crashlooping pods clears the alerts and leaves every silently degraded pod in place. Drain the node.

Escalate to GCP when the replacement node is also broken, or when a fleet scan shows several failing nodes. That points at VPC, firewall, or platform rather than one VM.

Root cause

The VM was reclaimed before anyone could inspect it, so no single cause was proven for the 2026-08-08 incident. Two hypotheses survive.

Node-local Dataplane V2 SNAT or masquerade state failure fits best. Public egress needs state that internal traffic does not, so a failure there preserves pod-to-Service and pod-to-apiserver traffic while killing everything outbound.

A GCP per-instance external-IP NAT fault fits almost as well. The node carried its own external IP, and a fault in Google’s translation path leaves the guest healthy. This one is worth a support case, since it may only appear in Google’s telemetry.

Graph View

Backlinks