Plastic Labs Internal Wiki

Recently Updated

Releasing a Honcho version onto Groudon

4 min read

Cutting the Honcho version comes first — Honcho release process. The git tag builds honcho:deployment-v<ver> in both Artifact Registries and registers the version with Groudon. Nothing runs it until secrets are loaded and available is flipped here.

This page is the release itself: making a version available to Groudon, then moving instances onto it.

Before you start

The management scripts live in groudon/groudon/scripts/ and take --test or --prod to pick the database. They read connection details from the repo .env; see connecting to the orchestrator DB for PROD_DB_URI and the percent-encoding trap.

Secrets are in AWS Secrets Manager, not the Supabase vault. Path layout, which the ExternalSecret template reads directly:

honcho/secrets/honcho-production-saas/<version>    # production
honcho/secrets/staging/<version>                   # staging

You need AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION=us-east-1 in your environment; the credentials are in the honcho-production-saas 1Password.

1. Load the secrets for the new version

A version’s secret is a JSON blob of everything Honcho needs. New versions copy from the latest existing one and apply a diff:

# new version, copying from the latest available and applying changes
uv run python -m groudon.scripts.load_honcho_secrets \
  --prod --new-version 3.1.0 --changes new_secrets.json
 
# amend an existing version in place
uv run python -m groudon.scripts.load_honcho_secrets \
  --prod --base-version 3.1.0 --new-version 3.1.0 --changes db_update.json

--changes takes {"add": {...}} (and removals). Do this before anyone boots the version, or the first instance on it comes up without its secrets.

2. The git tag registers the version

Pushing v3.1.0 (see Honcho release process) builds honcho:deployment-v3.1.0 and POSTs {version, image_label} to Groudon’s /webhooks/v1/add_honcho_version. That upserts a honcho_versions row. A new version defaults to available=False, has_migration=True.

available=False means tenants cannot upgrade to it yet. has_migration=True is the default assumption that this version carries a schema change, so auto-upgrade will not walk onto it.

The tag is the registration. When the version is ready, flip available to TRUE on that row (and has_migration to FALSE if it has no migration):

uv run python -m groudon.scripts.release_honcho --prod --version 3.1.0
# no schema change in this version: also clear has_migration so auto-upgrade may use it
uv run python -m groudon.scripts.release_honcho --prod --version 3.1.0 --no-migration

The script refuses if the tag has not registered the row yet, shows the current values, and asks for confirmation before the UPDATE.

At this point the image exists and Groudon knows about it. Nothing is running it.

3. Watch the rollout

A version bump only records desired state. The instance is still serving the old version when the write returns. The chain is:

honcho_instances.version bumped
   └─ DB trigger → groudon-health render → commit to argo-production
        └─ ArgoCD sync → rolling update of app-api / app-deriver
             └─ health observes it serving the target version → UPDATING → LIVE

So a released tenant sits in UPDATING for a while by design. The bump claims an upgrade lease rather than stamping the state directly; health’s _finalize_k8s_upgrade_if_rolled_out closes it. A tenant stuck in UPDATING with no open lease is a bug, and it logs exactly that.

SELECT count(*), version, instance_state
FROM groudon.honcho_instances hi
JOIN groudon.tenants t ON t.id = hi.tenant_id
GROUP BY version, instance_state ORDER BY 2, 3;

If the count is not moving, the pipeline is the suspect, not the release — see the GitOps pipeline.

Auto-upgrade

Tenants also move on their own, from the upgrade maintenance job in groudon-health (every 5 minutes, MAX_AUTO_UPGRADES_PER_RUN at a time, gated on AUTO_UPGRADE_ENABLED).

The rule: same major only, patch and minor both eligible, and the walk stops below the first version that has a migration. A major bump carries breaking API changes and is always scheduled by hand.

current 3.0.1; available 3.0.2, 3.1.0, 3.2.0 (has_migration), 4.0.0
  → upgrades to 3.1.0

Retagging does not work on production

If you need to fix a released version, cut a new tag. Prod sets no imagePullPolicy, so it defaults to IfNotPresent and any node that already cached deployment-v3.1.0 keeps serving the old image forever. Staging sets Always and does pick up a retag on restart, which makes this trap easy to miss — it works when you test it.

Other scripts worth knowing

  • scale_tenant_deployment — scale one tenant’s app-api / app-deriver
  • find_tenants_missing_instances — tenants that should have an instance and do not
  • create_k8s_triggers — re-assert the allocation and render DB triggers

The full list, including the local-development scripts and their safety rails, is groudon/groudon/scripts/README.md. update_honcho_instances no longer exists — the render pipeline does what it used to do.

See also

Graph View

Backlinks