Where things run, what they cost, and how to reach them.
Hardware (on-prem)
| Codename | Box | Specs | Used for |
|---|---|---|---|
| Porygon | Tinybox V1 Green | 6× RTX 4090 (~40GB each) | Inference (Metagross), benchmark harnesses (Excadrill), trace generation (Minccino) |
| Varoom | DGX Spark | NVIDIA Spark | Training (Machamp), heavier ML workloads |
Both are reachable through Tailscale — they don’t have public IPs by design. See Raikou for the Tailscale-based proxy used to route LLM inference through these machines.
VRAM/OOM is a recurring constraint — see Metagross’s parallelism strategies for how we partition the Tinybox.
Cloud
GCP (everything customer-facing)
| Service | Used for |
|---|---|
| GKE Autopilot | 46 tenant clusters (cluster-0…cluster-45) plus one operations cluster |
| AlloyDB | Groudon’s orchestrator DB, and one database per Honcho instance, across ~20 HA clusters |
| Cloud Run | cluster-manager, webhook-listen — cluster provisioning only. Tenant renders run inside groudon-health. |
| Cloud Build | Runs terraform for cluster creation |
| Artifact Registry | Container images for Honcho, Groudon, Xatu |
| Cloud DNS | The private prod.internal zone |
| Memorystore | Two Redis Clusters: Honcho + Groudon at redis.prod.internal (3 shards), Xatu dedup at redis.telemetry.prod.internal |
| Vertex AI | Production LLM serving for Anthropic Haiku and Gemini routes |
| GCS | Long-term artifacts (Machamp checkpoints, Excadrill datasets, terraform state) |
| Pub/Sub | Log transport out of the vector aggregator |
Everything on GKE is deployed by ArgoCD from git. See the GitOps pipeline.
AWS
- Secrets Manager — every Honcho instance’s secrets. External Secrets Operator pulls
them into clusters from an
aws-smClusterSecretStore. Paths are version-scoped, so a Honcho bump means a new secret path. - S3 — the Xatu Parquet archive.
Fly.io (residual)
Five apps left: prod-grafana, groudon-metrics-backup (Sableye),
tentacruel-litellm-prod, honcho-bot, and falinks (one Machine, no public ports —
see Internal apps). No Honcho tenants,
no Groudon, no Xatu compute.
Cloudflare
- Kyogre frontend on Cloudflare Pages, backend on Cloudflare Workers
- R2 for Kyogre artifact / theme / identity / marketplace storage
- KV for Kyogre rate limiting (eventually consistent)
- Cloudflare Access gates the internal apps —
triage.plasticlabs.ai(a Worker) andfalinks.plasticlabs.ai(a Fly Machine reached over a Cloudflare Tunnel). See Internal apps
Modal
- Serverless GPU option for Metagross deployments where Tinybox capacity is constrained
- Also fronts some model routes; a preemption there has taken out Honcho LLM calls before
Networking
The 10.x plan
One VPC, prod-plastic-labs-network, in us-east4. The second octet tells you what an
address is.
| Range | What |
|---|---|
10.0.0.0/20 | database-subnetwork (us-central1, legacy) |
10.1.0.0/16 | Private Service Access, allocated for AlloyDB |
10.125.0.0/21 | proxy-only subnet — GKE needs this for internal load balancers |
10.127.0.0/18 | operations subnet — Groudon, ArgoCD, Xatu, metrics |
10.128.0.0/21 … | tenant clusters, one /21 each: 10.(128 + N) for cluster-N |
10.174.0.0/24+ | do not use — orphaned PSA subnets in Google’s project |
Each cluster /21 also has two extra ranges Kubernetes uses, not the primary subnet:
pods at 10.x.64.0/18 and Services at 10.x.32.0/19.
10.174 is a real hole. Deleted AlloyDB clusters left three /24s in Google’s producer
project. They are invisible from our VPC, but creating a subnet there fails with
“overlaps with an active peer network”. The allocator sets reserved_octets = [174],
which is why cluster-46 sits on 10.175.
How managed services reach the VPC
Two mechanisms. Which one a service uses decides what IP it can have.
- Private Service Access peers Google’s VPC to ours and draws from a range we
allocate (
10.1.0.0/16). Peering cannot overlap, so a PSA-backed service can never sit inside one of our own subnets. AlloyDB works this way. - Private Service Connect puts a forwarding rule in our subnet pointing at
Google’s service. So a PSC-backed service gets an ordinary
10.127.xaddress. Memorystore works this way, via a service connection policy that names theoperationssubnet.
Names
prod.internal is a private Cloud DNS zone. Terraform creates the per-cluster records;
everything else is by hand — Adding an internal service is the
procedure.
| Name | Points at | Address held by |
|---|---|---|
cluster-N.prod.internal | that cluster’s control plane | terraform |
*.cluster-N.api.prod.internal | that cluster’s GKE Gateway, then its edge-proxy | terraform |
orchestrator.db.prod.internal | Groudon’s AlloyDB primary | PSA |
redis.prod.internal | Memorystore Redis Cluster discovery endpoint | PSC, auto |
redis.telemetry.prod.internal | Xatu’s Memorystore cluster | PSC, auto |
mimir.metrics.prod.internal | Mimir’s internal load balancer | ephemeral ILB IP |
vector-aggregator.log-ingest.prod.internal | the log sink | ephemeral ILB IP |
xatu-ingestion.xatu.prod.internal | Xatu ingestion | reserved xatu-ip-production |
The two marked ephemeral are ILB addresses that were never reserved, so recreating the Service changes the address and silently makes the record stale. Look them up live rather than trusting this table.
Staging uses prod.internal too. The plastic-labs-staging project holds a zone
named prod-internal serving prod.internal., and that is the one with records in it.
The staging-internal zone serving staging.internal. is empty — nothing resolves
there. The same hostname means different addresses in different projects, and
--project is what disambiguates.
Internal access
Tailscale / WireGuard mesh for anything Pokemon-named without a public DNS record, including the GPU boxes. Raikou is the Tailscale-based proxy for routing LLM inference to Porygon and Varoom.
Cloudflare Access for internal apps that do have a public hostname — Google SSO
restricted to @plasticlabs.ai. This is the whole access-control layer for those apps,
so any hostname that bypasses Cloudflare’s edge bypasses authentication entirely. See
Internal apps.
Telemetry stack
Two pipelines. Each tenant cluster runs an agent that ships to the operations cluster.
| Layer | Tool | Notes |
|---|---|---|
| Metrics collection | vmagent | One Deployment per cluster. Scrapes locally, aggregating on the way out, remote-writes to Mimir |
| Metrics storage | Mimir | metrics namespace, operations cluster |
| Dashboards | Grafana | Self-hosted beside Mimir |
| Log collection | vector | One DaemonSet per cluster (a copy on every node) |
| Log transport | vector-aggregator → Pub/Sub | Aggregator in log-ingest on operations |
| Event archive | S3 | CloudEvents → Parquet via Xatu |
| LLM proxy/telemetry | Tentacruel (LiteLLM proxy) | Wraps every LLM call for usage/cost |
| Errors | Sentry | |
| LLM traces | Langfuse | |
| Product analytics | PostHog |
Approximate run-rate (snapshot)
| Service | ~Cost | Notes |
|---|---|---|
| GCP | ~$27K/mo | Vertex AI (Haiku/Gemini) is the bulk at ~$25K |
| Modal | ~$2.3K/mo | Ramping |
| Tinybox V1 Green | $22.8K capex | Porygon |
| DGX Spark | ~$4.1K capex | Varoom |
This predates the GKE cutover and the compute-class work, which cut node count from 101 to 56 across 15 clusters. Treat it as stale.
Procurement preference: pay-as-you-go / self-serve cloud billing over enterprise contracts.
Model routing (production)
| Honcho agent | Model |
|---|---|
| Deriver | gemini-2.5-flash-lite (Vertex) |
| Dreamer | gemini-3.0-flash-preview (Vertex) |
| Dialectic | claude-haiku-4-5 (Vertex) / gemini-3.0-flash-preview (Vertex) |
| Minccino judge | gpt-4.1-mini |
| Local privacy filter | openai/privacy-filter (1.5B MoE, Tinybox) |
Provider abstraction is in [[llm-client-refactor|src/llm/]] — three native-SDK backends (Anthropic, OpenAI, Gemini) behind a ProviderBackend Protocol.
Instances reach the LiteLLM proxy over the public internet today, via BASE_URL env vars in
the version-scoped AWS secret. Moving it onto the VPC is tracked in DEV-2102.
Compliance posture
SOC 2 status, trust center, and legal thresholds live in Security & Compliance.