Plastic Labs Internal Wiki

Recently Updated

Adding an internal service

2 min read

A new service on the operations cluster (or a managed GCP service) that other things in the VPC need to reach by name. Terraform creates the per-cluster records. Everything else is by hand.

The names that already exist are in Infra. This page is how to add one.

Staging also uses prod.internal

The plastic-labs-staging project has a zone named prod-internal serving prod.internal. — that is the zone with records in it. staging.internal is empty. A staging config that says redis.prod.internal is correct; --project is what disambiguates. This is a relic of the migration to GKE, which was never corrected to point our staging project to a staging.internal zone.

For anything new: reserve an IP, then name it

Pick a free address in the operations subnet (10.127.0.0/18) — see what is occupying an IP range. Reserve it first, then pin the Service to it, so deleting and recreating the Service does not change the address.

gcloud compute addresses create <service-name> \
  --project=plastic-labs-prod --region=us-east4 \
  --subnet=operations --addresses=10.127.63.61

Pin the Service (networking.gke.io/load-balancer-type: "Internal", type: LoadBalancer, loadBalancerIP: "10.127.63.61"). Then:

gcloud dns record-sets create <service-name>.<namespace>.prod.internal. \
  --project=plastic-labs-prod --zone=prod-internal \
  --type=A --ttl=300 --rrdatas=10.127.63.61

Check from inside the VPC, not your laptop:

kubectl --context gke_plastic-labs-prod_us-east4_operations -n <ns> \
  run dnscheck --rm -it --restart=Never --image=busybox:1.36 -- \
  sh -c 'nslookup <service-name>.<namespace>.prod.internal'

Then point the consumer at the name. A pod holding the old value keeps it until it restarts — if that value is in a secret, restarting the fleet is a roll (roll-secret.sh), not a config change.

Add the name to the table in Infra. A *.prod.internal record that only lives in someone’s shell history is how vector-aggregator shipped nothing to BigQuery on production for months.

See also

  • Infra — the 10.x plan, PSA vs PSC, the name table
  • Where to look — what already occupies a range

Graph View

Backlinks