Plastic Labs Internal Wiki

Recently Updated

ArgoCD operations

2 min read

Upgrading, resizing and unsticking the ArgoCD control plane on the operations cluster. ArgoCD reconciles many applications across many tenant clusters.

Config lives in infra-operations/argocd/{production,staging}/values.yaml. That repo is not in the ArgoCD-managed set — ArgoCD cannot deploy itself, so these changes are applied by hand with Helm.

The mental model that matters

ArgoCD being down does not take production down. It is a reconciler: it compares git to the clusters and applies the difference. If it stops for ten minutes, every Honcho pod keeps serving traffic exactly as before. What you lose is new changes landing and drift being corrected.

So the risk during an upgrade is “changes queue up,” not “production breaks.” That is what makes a mid-day upgrade acceptable.

Upgrading

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
 
gcloud container clusters get-credentials operations \
  --region us-east4 --project plastic-labs-prod
 
helm upgrade --install argocd argo/argo-cd \
  --namespace argocd --version 10.2.2 \
  -f infra-operations/argocd/production/values.yaml \
  --atomic --create-namespace --timeout 45m

What each flag actually does:

  • --version 10.2.2 pins the chart, not ArgoCD. Chart 10.2.2 ships ArgoCD v3.4.6. Check with helm search repo argo/argo-cd --versions.
  • --atomic implies --wait — Helm blocks until every resource reports ready and rolls the whole release back on any failure. This is what you want, and it is why the timeout has to be generous.
  • --timeout 45m — the controller StatefulSet rolls one pod at a time and each new shard has to re-list every cluster it owns. Budget tens of minutes, not seconds.

The argocd release is a normal Helm release; nothing about it appears in the ArgoCD UI, because ArgoCD does not manage itself. Watch it with kubectl:

kubectl --context gke_plastic-labs-prod_us-east4_operations -n argocd \
  get pods -w

Credentials

admin, with the password in Secret Manager on the matching GCP project. On a fresh install:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

prod · staging

See also

  • GitOps pipeline — sync waves and the traps
  • Compute classes and spot — fleet-wide nodeSelector changes hit the control plane hard
  • infra-operations/argocd/README.md — first-time install, Workload Identity bindings, cluster registration

Graph View

Backlinks