deriver-polling-resilience

Deriver Polling Resilience: Remove Connection-Acquisition Retry + Add Poll Jitter

Status: Draft | Owner: vineeth | Last updated: 2026-06-02 Scope: repos/honcho — deriver (src/deriver/queue_manager.py) + DB session layer (src/db.py, src/config.py)

---

1. Problem

On 2026-06-02 the managed (Groudon) shared Postgres suffered a ~4.5-hour connection-saturation outage (Supavisor client connections 300 → ~9,000, EMAXCONN; pooler avalanche onset ~02:10 ET). The pg_cron connection-state samplers (public.mon_*, see repo-root lock_monitoring.sql) captured the onset and conclusively identified the mechanism. It is not lock contention:

• Synchronized poll herd. Server-side active backends sat at ~15–25, then slammed to 251 of ~368 (the pool ceiling) in a single second (01:59:26), repeating in bursts. Every burst was dominated by the deriver queue-claim query (SELECT … work_unit_key FROM <schema>.queue WHERE NOT processed …) firing across many distinct tenant schemas at the same instant.
• Not locks. advisory waits were ~0 for the entire onset; pg_locks peaked at 2,272 vs a 32,000 ceiling (max_locks_per_transaction=64 × ~500). Both lock hypotheses are ruled out for this event.
• Sustained, not transient. The collapse held for ~4 hours and only cleared when overnight traffic fell / after a full DB restart — it did not self-recover.

There are ~11,543 tenant instances (367 on v3.0.8, 11,176 on v3.0.7), each running its own deriver. At steady state, ~11.5k derivers polling ~every 30s averages ~65 concurrent claim queries — well within the 368-backend pool. The failure is variance, not throughput: the same work bunched into one second instead of spread across thirty.

2. Root Cause — two cooperating flaws

Trigger — no jitter in the poll schedule. _advance_poll_interval (src/deriver/queue_manager.py:~395) advances deterministically (min(interval × POLLING_BACKOFF_MULTIPLIER, POLLING_SLEEP_MAX_INTERVAL_SECONDS)), and the base-interval sleeps are a fixed POLLING_SLEEP_INTERVAL_SECONDS. When a cohort of derivers starts together — a rollout batch, or every instance after the DB restart used to recover the prior event — their poll loops phase-align and stay aligned, producing the synchronized herd. The recovery action (full DB restart) re-synchronized the entire fleet and seeded the next collapse.

Sustain — connection-acquisition retry on the deriver path. Each poll acquires a connection via HonchoAsyncSession → acquire_connection_with_retry (src/db.py:110), which retries on OperationalError (Supavisor EMAXCONN) with jittered exponential backoff up to CONNECTION_RETRY_MAX_DELAY_SECONDS = 10s (src/config.py:639) before giving up. The poll loop’s own “back off a saturated DB” handler (the except → _advance_poll_interval, queue_manager.py:~452) only runs after that 10s. So during saturation each poll cycle:

1. fires, can’t get a slot,
2. holds a pending client connection and re-knocks for up to 10s,
3. only then lets the poll loop back off.

With thousands of derivers doing this, the retry manufactures the multi-thousand client-connection wall and prevents the pool from draining: as fast as a backend frees, a waiting retrier grabs it. Retrying is correct for a transient blip; it is exactly wrong for sustained saturation, where the system must shed load, not persist demand. This is what converted a ~30-second herd into a 4-hour outage.

Clean separation of the two graphs: the active-backend spike = the unjittered poll herd; the client-connection wall + non-drainage = the connection retry.

3. Goals / Non-Goals

Goals

• G1: Desynchronize the deriver fleet so a mass restart (rollout or DB restart) cannot phase-align polls into a herd.
• G2: Make connection acquisition fail fast everywhere — a single attempt, no server-side retry — so a saturated pooler is never hammered by its own clients holding connections open to re-knock.
• G3: On the API path, surface acquisition failure as a fast error (503 recommended) and let retry happen in the client/SDK layer, where it can back off without pinning a server-side connection. On the deriver path, the existing jittered poll backoff absorbs it.
• G4: No schema migration — ship as a migration-less patch that the Groudon auto-upgrader rolls automatically.

Non-Goals

• NG1: Queue partial-index optimization (cuts the 170ms claim cost) — separate ticket; complementary.
• NG2: A full API-path circuit breaker / load-shedding layer (concurrency caps, half-open probing) — separate ticket. This spec does the minimal correct thing (fail fast + a clean 503); a richer breaker is an enhancement on top.
• NG3: pg_stat_statements.max / schema-qualified-SQL normalization — separate.
• NG4: The advisory-lock removal (message_seq_counter, DEV-1852) — confirmed not on this incident path; unchanged here.

4. Change 1 — Jitter the deriver polling

Two layers; the startup jitter is the high-value one for the herd.

4a. Startup jitter (most important). Before the first poll, sleep a uniform-random delay in [0, POLLING_SLEEP_MAX_INTERVAL_SECONDS]. A mass restart then spreads first-polls across the whole window instead of all firing at t=0.

# queue_manager.py, before entering polling_loop()
import random
await asyncio.sleep(random.uniform(0.0, settings.DERIVER.POLLING_SLEEP_MAX_INTERVAL_SECONDS))

4b. Per-cycle jitter. Jitter every sleep so loops that drift toward alignment are continuously re-scattered. Keep the underlying backoff schedule deterministic (so the 1→2→4→…→30 progression is unchanged); jitter only the returned sleep.

def _advance_poll_interval(self) -> float:
    interval = self._current_poll_interval
    if settings.DERIVER.POLLING_BACKOFF_ENABLED:
        self._current_poll_interval = min(
            self._current_poll_interval * settings.DERIVER.POLLING_BACKOFF_MULTIPLIER,
            settings.DERIVER.POLLING_SLEEP_MAX_INTERVAL_SECONDS,
        )
    return self._jitter(interval)
 
def _jitter(self, seconds: float) -> float:
    j = settings.DERIVER.POLLING_JITTER_RATIO       # e.g. 0.5 → sleep in [0.5x, 1.5x]
    return seconds * random.uniform(1.0 - j, 1.0 + j)

Apply _jitter(...) to the fixed base-interval sleeps as well (the semaphore.locked() branch and the empty-queue branch in polling_loop, queue_manager.py:~422/~445), so the “fast pickup” 1s path doesn’t re-synchronize the fleet either.

New config (src/config.py, DeriverSettings):

POLLING_JITTER_RATIO: Annotated[float, Field(default=0.5, ge=0.0, le=1.0)] = 0.5
# Startup jitter reuses POLLING_SLEEP_MAX_INTERVAL_SECONDS as its upper bound.

5. Change 2 — Remove the connection-acquisition retry entirely (fail fast everywhere)

Rip out the retry so every connection checkout is a single attempt. This is one global change — not a per-process .env toggle (which is operationally painful to apply across ~11.5k deriver machines and easy to get wrong). The deriver and the API both fail fast; they differ only in how the failure is surfaced (the deriver loop backs off; the API returns a fast error).

src/db.py — delete the retry:

• Reduce _ensure_acquired to a single await self.connection() (keep the existing Sentry span + metric for visibility), set _honcho_acquired, and let any OperationalError propagate. acquire_connection_with_retry collapses to that single checkout (or is inlined and removed).
• Delete the tenacity imports (AsyncRetrying, retry_if_exception_type, stop_after_delay, wait_exponential_jitter, src/db.py:16), RETRYABLE_DB_CONNECTION_ERRORS (:68), and the retry loop (:132–162). Keep a simplified _record_acquisition_outcome("ok"|"failed") if you want the metric.
• Bounded connect timeout so the single attempt fails fast instead of hanging when the pooler is saturated:

  # src/db.py
  connect_args = {"prepare_threshold": None, "connect_timeout": 2}  # psycopg seconds

src/config.py — remove the now-dead knobs: CONNECTION_RETRY_ENABLED, CONNECTION_RETRY_MAX_DELAY_SECONDS, CONNECTION_RETRY_BACKOFF_INITIAL_SECONDS, CONNECTION_RETRY_BACKOFF_MAX_SECONDS (:638–646), and the _validate_retry_budget_vs_pool_timeout validator. (POOL_TIMEOUT stays but only applies to a real QueuePool; the managed deployment runs NullPool.)

API path — fast failure, correct status. With the retry gone, an OperationalError propagates to the existing global handler (src/main.py:234), which already returns 500 fast — so the “send a 500 quickly rather than hammer the DB” behavior is the default the moment the retry is removed; no handler change is strictly required. Recommended refinement: add a dedicated handler mapping pooler saturation to 503 Service Unavailable with Retry-After, so SDK clients treat it as transient and back off client-side (where retry belongs) rather than as a hard 500:

from sqlalchemy.exc import OperationalError
 
@app.exception_handler(OperationalError)
async def db_unavailable_handler(_request: Request, exc: OperationalError):
    return JSONResponse(status_code=503, headers={"Retry-After": "1"},
                        content={"detail": "database temporarily unavailable"})

Deriver path — unchanged, just faster. The poll loop’s existing except (queue_manager.py:~452) catches the now-immediate OperationalError and backs off via the jittered _advance_poll_interval from Change 1. No deriver-specific code or env needed.

6. Why failing fast is safe — both paths

• Deriver: the queue is the durability boundary. A poll that can’t acquire a connection claims nothing; a task that fails mid-flight leaves its QueueItem unprocessed. The work stays enqueued and is retried on a later (jittered) cycle — no data loss, at most a brief delay.
• API: during sustained saturation the retry does not succeed — it only defers the same error by up to 10s while holding a client connection and feeding the collapse. A fast 503/500 returns the connection immediately so the pool can drain, and lets the caller’s SDK retry with its own backoff (client-side retry doesn’t pin a server connection). A fast error during a genuine outage is strictly better than a slow one that extends the outage. Retrying is the right response to a transient blip and the wrong response to sustained saturation — and the server can’t distinguish them at acquisition time, so it shouldn’t try. Server-side retry is also redundant with the Honcho SDKs’ own client-side retry/backoff.

7. Rollout

• Migration-less, code/config only → qualifies for the Groudon auto-upgrader (no has_migration gate). Ships as a normal patch.
• Operational guardrails for this rollout specifically (the rollout is itself a herd source):
  • Stagger upgrade batches (already 20 / 5 min) and rely on the new startup jitter to desync restarted instances.
  • Do not recover by full DB restart. A full restart re-synchronizes the entire fleet and re-arms the herd. Prefer draining / pausing new client acceptance, or restart in staggered cohorts. Capture this in the Groudon incident runbook.
• Verify post-deploy via the mon_* tables: the per-second active series should no longer show synchronized 200+ spikes; client connections should track average load, not staircase.

8. Testing

• Jitter desync (unit): N loops started simultaneously produce first-poll times spread across [0, MAX], not clustered; _advance_poll_interval returns values within [1−J, 1+J]×schedule while the underlying schedule still progresses 1→2→…→30.
• Fail-fast (integration): with a pooler returning EMAXCONN, a connection checkout raises within ~connect_timeout, not ~10s. Deriver: the poll loop catches it and backs off. API: the request returns a fast 503 (or 500) — assert sub-second latency, not multi-second.
• Single attempt: under a failing pooler, exactly one checkout is attempted per poll/request — no repeated connection attempts (the regression guard against re-introducing retry).
• No work loss: items unclaimed during a simulated saturation window are processed on a later cycle once capacity returns.
• Fleet simulation (optional): many simulated derivers + a capped pool — confirm jitter flattens the concurrent-claim distribution vs. the deterministic baseline.

9. Risks & Alternatives

• Risk: startup jitter delays first work pickup by up to 30s after deploy. Acceptable for a background worker; bound it lower (e.g. [0, base×k]) if pickup latency matters.
• Risk: removing API retry means a transient single-connection blip now surfaces as a client-visible 503/500 instead of being silently ridden out. Accepted — the Honcho SDKs retry with backoff client-side, and server-side retry under saturation is net-harmful. The blast radius of a brief blip (a few retryable 503s) is far smaller than a multi-hour collapse.
• Alternative — keep a short (e.g. 1s) bounded retry. Rejected: any server-side retry under sustained saturation re-feeds the collapse, and the 10s budget already proved harmful in production. “Single attempt” is the clean, defensible invariant; tuning a budget invites the same failure mode back.
• Alternative — global in-flight acquisition cap / circuit breaker. Stronger systemic guard but cross-cutting; deferred to the NG2 follow-up.

10. Out of Scope (tracked separately)

• Queue partial index on unprocessed representation work units (claim-query cost).
• API-path circuit breaker / load-shedding.
• pg_stat_statements.max increase + schema-qualified-SQL normalization.
• message_seq_counter advisory-lock removal (DEV-1852) — not on this incident path.