Compute classes and spot

Every Honcho pod on the tenant fleet is pinned to a custom ComputeClass — a GKE Autopilot object that tells the autoscaler which machine types to create nodes from, in what order. The classes are defined once in argo-source/base_networking/templates/computeClass.yaml and selected per workload with a nodeSelector.

This page is why the classes are shaped the way they are, and what to know before changing them.

The three classes

• n4-n2d-any — general. n4 then n2d, any size.
• n4-n2d-highmem-110 — prod app-api. n4-highmem-8 → n2d-highmem-8, 110 pods/node, on-demand.
• c3-n2-highmem-110-spot — prod app-deriver. c3 → n2 → c2d highmem-8 on spot, falling back to on-demand n4.

Why maxPodsPerNode: 110

A Honcho pod requests 63m CPU / 500Mi memory / 128Mi ephemeral. At 110 slots minus ~15 DaemonSet pods a node can hold 95 of them, so the binding constraint is memory, not CPU:

shape                $/mo   pods/node   $/pod-month
n4-standard-8 @64     265       49          5.40
n4-standard-8 @110    265       53          5.00
n4-highmem-8  @110    348       95          3.66

highmem is $83/mo more per node. The saving comes only from needing fewer of them, so it is a rounding effect — a cluster with fewer than ~110 pods would be cheaper on standard-8.

Why the deriver runs on spot and the API does not

The deriver is a queue worker. It handles SIGTERM, and a work unit abandoned by a hard kill is reclaimed by cleanup_stale_work_units after STALE_SESSION_TIMEOUT_MINUTES. A preemption therefore costs one session a few minutes of derivation latency — not data.

The API cannot tolerate that. A preemption mid-request is a 5xx to a customer. It stays on on-demand n4.

That asymmetry is the whole design, and it is why the two workloads want different machine families.

Why c3 for spot and n4 for on-demand

Verified against the Cloud Billing catalog (service 6F81-5844-456A, us-east4), highmem-8, $/node-month, August 2026:

• c3 — on-demand 386.34, spot 98.68 (74.5%) ← spot floor
• n2 — on-demand 430.83, spot 108.50 (74.8%)
• c2d — on-demand 402.73, spot 128.67 (68.1%)
• n4 — on-demand 347.54, spot 187.27 (46.1%) ← on-demand floor
• n2d — on-demand 374.83, spot 224.95 (40.0%)

n4 is the cheapest on-demand highmem-8 and nearly the worst on spot. c3-highmem-8 is the same 8 vCPU / 64 GB geometry for 47% less on spot. So the spot ladder is ordered strictly cheapest-first and its fallback floor is n4 on-demand, not c3 on-demand — n4 is $44/mo cheaper there.

Ordering priorities cheapest-first is also Google’s documented workaround for the autoscaler comparing spot prices against a us-central1 baseline.

activeMigration is load-bearing

activeMigration:
  optimizeRulePriority: true

Without it GKE never moves a workload back up the ladder — a deriver that fell back to on-demand n4 during a spot shortage would stay there indefinitely. From Google’s docs: “If the activeMigration.optimizeRulePriority field is set to false or omitted, GKE won’t automatically move workloads to higher-priority nodes when they become available.”

Quota

Spot draws on PREEMPTIBLE_CPUS, a separate quota from your on-demand CPU quota. In us-east4 it is 10,000 against ~0 prior usage; the fleet needs ~824 vCPU, which fits easily. Note the families are not interchangeable for quota either — we hold 500 c3D vCPUs but effectively unlimited c3A.

What it looks like when it is working

A converged production cluster carries both classes side by side:

kubectl --context gke_plastic-labs-prod_us-east4_cluster-33 \
  get nodes -L cloud.google.com/compute-class,cloud.google.com/gke-spot

3  c3-n2-highmem-110-spot   true     <- derivers
3  n4-n2d-highmem-110                <- api
2                                    <- default pool (system)

Rolling out a change to the classes

Changing a nodeSelector in argo-source/base/templates/ recreates every pod it selects, fleet-wide. The pattern that worked for the spot migration:

1. Canary one cluster. Ship the class change behind an allowlist keyed on $.Values.clusterset (the same shape as the privateNodeAllowList still in honcho-deriver.yaml), containing a single cluster.
2. Watch it converge — nodes created, pods Running, restart counts flat. Give it long enough to see a preemption, not just a rollout.
3. Widen to a handful, then remove the allowlist in a separate PR.

See also

• ArgoCD operations — what a fleet-wide rollout does to the control plane
• Infra — the cluster and IP plan
• GitOps pipeline — Autopilot’s resource-request rewriting under the Default class