Outgoing: Aakash (primary), Eugene (secondary). Incoming: Vineeth (primary), Eri (secondary).
Still open
DEV-2464 — Groudon and Argo both write spec.replicas
DEV-2464 · High · Todo · Aakash
The chart renders replicas: 1 only when instance_state == "live", else 0. groudon.k8s_ops also scales imperatively (cold-storage restore, billing pause, errored repair). Argo snapshots manifests at op start and applies them wave by wave. On 2026-08-24 an op started against a cold_storage render; groudon restored the tenant to 1; Argo then applied replicas: 0. The health loop’s UPDATING branch has no timeout, so a dead API stayed masked for ~50 minutes.
ignoreDifferences on /spec/replicas plus RespectIgnoreDifferences=true did not prevent the write on apply. Who patched replicas is in the GKE audit log (cloudaudit.googleapis.com/activity, filter protoPayload.request.spec.replicas); callerSuppliedUserAgent is groudon (OpenAPI-Generator) vs Argo (argocd-application-controller).
If a live tenant is suddenly at 0 replicas, check that audit log before assuming groudon meant it. Durable fix is still unpicked: either groudon re-asserts replicas after every Argo sync, or drop replicas from the chart and make groudon the sole writer.
Webhook deletion in groudon-web
The deletion case in groudon-web needs a patch. This is a real broken path, not “Sentry is noisy.” If the alert is still firing, that is why.
DEV-2313 — Detect silently-broken nodes
DEV-2313 · High · In Progress · Rajat
Pods stay Running and pass every probe while the node has no public egress. Sentry cannot see it (ingest is itself egress). First incident 2026-08-08, ~7 hours silent. Cordon + drain; do not just delete the crashlooping pods.
Related: Node loses egress.
The ticket is the detector (canary DaemonSet and/or node_flow external-packet ratio). Until that ships, discovery is still coincidence (a new pod crashlooping on tiktoken download).
Scope-backfill deriver memory
A tenant OOM’d while the deriver backfilled a large session (~14k documents) at the 1000Mi default.
- Tenant patch (shipped): argo-source#53 —
hchffo8lrz2u18qlv336ah2gderiver limit raised to 3Gi via anapp_name-keyed override. Other tenants unchanged. - Root fix (open): honcho#1104 (
fix/scope-backfill-memory) — bound the backfill so the override can come out. Review: Linear.
If another tenant OOMs on deriver during a big backfill, the override map is the same lever; do not raise the fleet default.
Resolved this rotation — still worth watching
DEV-2470 — Argo CD controller shards pack onto one operations node
DEV-2470 · Done
The prod - Node CPU allocatable > 90% policy fired on the operations cluster while cluster-wide utilization was ~18%. Four of ten argocd-application-controller shards (plus applicationset + repo-server) landed on one c4-highmem-8 after a node rotation. Requests were 1 CPU / shard, usage ~1.6, so the scheduler packed them.
Rajat raised controller CPU/mem (infra-operations@75fa305). No pages since; marked Done. Recurrence signature: operations node at 100% CPU, cluster-level still looks fine, alert node name __missing__ (policy groups by cluster_name only).
Argo CD on operations is still a hand-installed Helm release; values are not fully in git. helm -n argocd get values argocd is the live copy.