AlloyDB connection saturation
Tenant requests start failing with a login error while the database sits almost idle. The cluster has run out of connection slots.
Related: node-egress-failure · Tenant pod crashlooping
The error
Honcho surfaces it as an OperationalError from SQLAlchemy:
(psycopg.errors.ProtocolViolation) server login has been failing, cached error:
remaining connection slots are reserved for roles with the SUPERUSER attribute
The AlloyDB side logs it as ALERT severity, once per rejected login:
db=hch9dv9yvw1w7tvkgoz1bpfd,user=postgres FATAL: [postinit.c:1086]
remaining connection slots are reserved for roles with the SUPERUSER attribute
Why the pooler does not save you
Every cluster runs AlloyDB managed connection pooling in transaction mode. That pooling works inside one (user, database) pair and cannot cross databases.
A PostgreSQL backend picks its database in the connection startup packet. The protocol has no message to switch databases afterwards, so a backend is bound to one database for its whole life. The pooler can multiplex thousands of clients onto a handful of backends for a single tenant. It can never hand a backend for tenant A to tenant B.
You run one database per tenant. So the floor on real backends is:
backends ≈ number of tenant databases receiving traffic × 1 to 2 each
Derivers poll their tenant database periodically. The pooler closes an idle server connection after server_connection_idle_timeout, which defaults to 600 seconds. A poll every 30 seconds resets that timer, so a live tenant’s pool never drains.
A measured example from 2026-08-08:
983 backends of 1000 max_connections
2 active
964 idle
31 belonging to alloydbadmin
Almost a thousand connections, two queries running. Nothing is leaking. Each live tenant is holding its own warm pool, and the pooler is doing its job.
Reading the numbers
From inside the database, using the postgres role:
-- headline numbers
SELECT count(*) AS backends,
current_setting('max_connections') AS max_conn,
count(*) FILTER (WHERE state = 'active') AS active,
count(*) FILTER (WHERE state = 'idle') AS idle
FROM pg_stat_activity;
-- per-pool depth. Each tenant database is one pool.
SELECT datname, usename, count(*), max(now() - state_change) AS oldest_idle
FROM pg_stat_activity
WHERE datname LIKE 'hch%'
GROUP BY 1, 2 ORDER BY 3 DESC LIMIT 25;
-- how stale the idle backends are. Buckets near zero mean polling keeps them warm.
SELECT width_bucket(EXTRACT(epoch FROM now() - state_change), 0, 600, 6) AS bucket_of_10min,
count(*)
FROM pg_stat_activity WHERE state = 'idle'
GROUP BY 1 ORDER BY 1;
-- everything should be pooler-owned loopback
SELECT client_addr, count(*) FROM pg_stat_activity GROUP BY 1 ORDER BY 2 DESC;The idle-age histogram is the one that explains the whole situation. A healthy-looking distribution has most connections used within the last 100 seconds, which shows they are busy in short bursts rather than abandoned.
The gap between your setting and your hardware
Context for the 2026-08-08 incident. The clusters ran max_connections = 1000, which was never chosen. It is AlloyDB’s default.
Google’s recommended value depends on the machine:
| vCPU | Memory | Recommended max_connections |
|---|---|---|
| 1 | 8 GB | 500 |
| 2 | 16 GB | 1000 |
| 4 | 32 GB | 2000 |
| 8 | 64 GB | 4000 |
| 16 | 128 GB | 5000 |
| 32+ | 256 GB+ | 5000 |
Production clusters run c4a-highmem-8-lssd, meaning 8 vCPU and 64 GB. The recommended value is 4000.
Check what a cluster actually has:
gcloud alloydb instances describe prod-alloydb-ha-NN-primary \
--cluster prod-alloydb-ha-NN --region us-east4 --project plastic-labs-prod \
| grep -A3 machineConfigor via the AlloyDB UI.
Mitigation
Raise max_connections to 4000. This is a database flag, so it restarts the instance. On a REGIONAL cluster that is an HA failover of roughly 20 to 60 seconds.
The console UI is the safer path. It presents existing flags as a form, so you add to them. The gcloud equivalent replaces the entire flag set, and omitting a flag silently deletes it:
gcloud alloydb instances update prod-alloydb-ha-NN-primary \
--cluster prod-alloydb-ha-NN --region us-east4 --project plastic-labs-prod \
--database-flags='<every existing flag>,max_connections=4000'Pull the current flags first and carry all of them forward.
What the operation looks like, measured on 2026-08-07:
19:59 update submitted
20:08 standby node starts up in the background
20:13 flag visible in the instance config, limit metric still 1000
20:15 connections_limit flips to 4000 ← the actual cutover, inside a 47s window
20:19 operation reports DONE ← about 19.5 minutes total
The instance reports READY with reconciling: true for most of that. Serving never stopped in any 45-second sample. The long tail is Google staging the change on a standby node and health-checking it before switching traffic.
After the flip, connections rebuild from zero as pools warm up again. Expect the count to settle slightly higher than before, because demand that was previously refused can now connect.
Verify:
Longer-term levers
Raising the ceiling buys room. It does not change the shape of the problem, which is that backends scale with the number of tenant databases being polled.
Stop quiet tenants from holding connections. A live tenant with an empty queue still polls every 30 seconds, which keeps its pool warm forever. Backing off past the 600-second idle timeout when the queue stays empty lets those pools drain to zero. This is the only change that lowers the floor rather than raising the ceiling.
Lower server_connection_idle_timeout. A pooler flag, applied live with no restart. It cannot touch the primary connection in each pool, since polling keeps that one fresh. It does reclaim the extra connections that pools accumulate during overlapping requests, which was roughly a quarter of the total on the measured cluster. Useful as a fast partial win, and testable on one cluster at a time.
Raising max_pool_size or adding clusters are weaker moves. Observed pool depth was about 2 against a limit of 6, so the cap is not binding, and new clusters spend money on a problem that placement fixes for free.