Plastic Labs Internal Wiki

Recently Updated

Deployment

7 min read

How a merged change reaches production, per service. This page covers the four things we ship on a regular cadence: Groudon, Xatu, Tentacruel, and Honcho.

The mechanics of ArgoCD, charts, batches, and sync waves live in the GitOps pipeline. This page is the layer on top of that: what triggers a build, what writes a new image tag, and how to debug when it goes wrong.

Top-level overview

Images land in GCP Artifact Registry (us-east4-docker.pkg.dev/plastic-labs-{staging,prod}/plastic-labs-{staging,production}-registry/). Staging typically deploys on merge to main. Production deploys are gated — a promotion to stable for Groudon, a manual image-tag PR for Xatu, a manual flyctl deploy for Tentacruel.

Groudon — gateway and health processes share the same docker image and run on the kubernetes operations cluster.

  • Staging: A merge to main builds groudon:<sha> and pushes the new image to GCP Artifact Registry on plastic-labs-staging. It then writes the new image tag into every Groudon manifest file in argo-staging. ArgoCD syncs the Groudon deployments with the new image.
  • Prod: Someone promotes main → stable. Same process as staging, but against argo-production.

Xatu — ingestion and consumer run as separate deployments on the operations cluster.

  • Staging: A push to main (in the Groudon repo) that touches xatu/** builds xatu:<sha> and pushes the new image to GCP Artifact Registry on plastic-labs-staging. It then writes the new image tag into argo-staging/xatu/{ingestion,consumer}.yaml. ArgoCD syncs the Xatu deployments with the new image.
  • Prod: Someone triggers a build via the GitHub Actions workflow (in the Groudon repo), which builds the image and pushes it to GCP Artifact Registry. Then they open a PR in argo-production that points at the new tag.

Tentacruel — the LiteLLM proxy, still on Fly.

  • Staging: merge to main deploys tentacruel-litellm.
  • Prod: someone runs flyctl deploy against tentacruel-litellm-prod.

Honcho — managed fleet. Merge and deploy are fully decoupled.

  • A git tag v* builds honcho:deployment-v<ver> into GCP Artifact Registry, for both staging and production, and registers the version with Groudon. Nothing runs it until someone releases it.

Groudon

Gateway and health deployments both run on the same docker image on the operations cluster. Their kubernetes manifests live in argo-production/groudon/ and argo-staging/groudon/.

Staging deployment

Merge to main.

  1. Builds and pushes a new groudon image tagged groudon:<sha> to GCP Artifact Registry on plastic-labs-staging.
  2. Checks out argo-staging and writes the new SHA into every Groudon manifest file that already references the image.
  3. Commits and pushes the changes to argo-staging.
  4. ArgoCD auto-syncs the gateway and health deployments with the new image, and runs the migrate-job to apply Alembic migrations.

Production deployment

Promotion from main to stable is manual. Once promoted, the production deployment follows the same process as the staging deployment.

Rolling back

Two options, and the fast one is not the git one:

  • Revert the manifest — PR argo-production/groudon/*.yaml back to the previous SHA. The image already exists in Artifact Registry; no rebuild, ArgoCD syncs in seconds.
  • Revert the code — revert on stable, which rebuilds and redeploys. Correct, slower, and the one to use when the bad commit must not stay on the stable branch.

Note that a migration that has already applied is not undone by either of these rollback options. A migration that has been applied must be manually reverted by running the alembic downgrade command for the specific migration.

Dashboard

The Groudon dashboard is deployed by Vercel, on merge to main or stable. It does not pass through ArgoCD.

Xatu

Lives in the Groudon repo under the xatu/ directory, but deploys separately from Groudon.

Xatu has two process groups — ingestion and consumer — and both run as their own kubernetes Deployment on the operations cluster in both plastic-labs-staging and plastic-labs-prod.

Two GitHub Actions are responsible for deploying Xatu to staging and production:

  • push-gcp-registry-xatu-staging.yml — on main, builds xatu:<sha> and writes that tag into xatu/ingestion.yaml and xatu/consumer.yaml in argo-staging.
  • push-gcp-registry-xatu-prod.yml — a manually triggered workflow. Builds and pushes the new image to GCP Artifact Registry on plastic-labs-prod, but production rollout requires a manual image-tag PR in argo-production.

Rollback is the same shape as Groudon: PR the argo-production/xatu/ manifest back to a previous SHA rather than reverting and rebuilding.

Tentacruel (LiteLLM proxy)

The LiteLLM proxy every Honcho LLM call goes through. Two Fly apps in iad, both performance-1x / 2 GB with a minimum of 3 machines:

  • tentacruel-litellm — staging. This is the app fly.toml declares, so a bare flyctl deploy targets it.
  • tentacruel-litellm-prod — production. Deployed with an explicit --app tentacruel-litellm-prod.

Shipping a change

Fly. Merge to main → fly-deploy.yml → flyctl deploy --remote-only. The workflow passes no --app, so it picks up fly.toml and deploys staging (tentacruel-litellm). Production (tentacruel-litellm-prod) is a deliberate manual deploy against that app — nothing automatic touches it.

Honcho

Honcho is the one service where merging and deploying are fully decoupled. The deployment unit is a git tag, and the rollout is a database write.

1. Tag

Pushing v3.4.5 fires both push-gcp-registry-staging.yml and push-gcp-registry-prod.yml. Each one:

  1. Strips the v and checks the tag against pyproject.toml’s version, failing the build on a mismatch. A running instance serves that version at /openapi.json, so the git tag and the version in pyproject.toml must agree. Bump pyproject.toml before tagging.
  2. Builds and pushes honcho:deployment-v<version>.
  3. POSTs {version, image_label} to <env>/webhooks/v1/add_honcho_version on Groudon’s API.

2. Register

Groudon’s gateway upserts a honcho_versions row for the newly tagged version. A new version defaults to available=False, has_migration=True. available=False means that no one can upgrade this version yet. has_migration=True means that this version contains a database migration (default assumption) so tenants will never auto-upgrade to this version.

After this step, the image exists and Groudon knows about it, but no tenants are running on this version. The release_honcho script in the Groudon repo is unused; the git tag is the registration.

3. Release

  1. Run load_honcho_secrets.py to write the version’s secrets to AWS Secrets Manager at honcho/secrets/{environment}/{version}. Paths are version-scoped, so every Honcho bump needs its own secret set. This step copies the secrets from the previous Honcho version, and uses a “changes” file to determine which new secrets to add/update/delete for the new version.
  2. Flip the available column to TRUE for the new version in the honcho_versions table. This makes the new version widely available to our tenants. Flipping the has_migration column on the new version to FALSE allows tenants to auto-upgrade to this version.

4. Roll out to tenants

All tenants individually upgrade to the new version. How this works:

  1. Either the tenant explicitly upgrades from app.honcho.dev or the upgrade maintenance job triggers an upgrade for a batch of tenants.
  2. These both write the new version to honcho_instances.version for the tenant’s honcho instance row.
  3. The rest happens asynchronously: the render job in groudon-health writes the tenant’s new yaml into argo-staging or argo-production/clusters/ and ArgoCD performs a rolling update of the app-api and app-deriver deployments.

Auto-upgrade (the upgrade maintenance job, runs every 5 min): when AUTO_UPGRADE_ENABLED is set to true, it upgrades LIVE, non-placeholder, non-cold-storage tenants along patch and minor versions of the same major, walking the sorted list and stopping at the first version with has_migration=True. A major bump is always scheduled by hand. At most MAX_AUTO_UPGRADES_PER_RUN (set to 20 in production) tenants per run.

See also

Graph View

Backlinks