Plastic Labs Internal Wiki

Recently Updated

Conclusion-Quality Eval Pipeline Overhaul

12 min read

Spec: Conclusion-Quality Eval Pipeline Overhaul

Status: Draft | Owner: vineeth | Linear: ML-335 | Prompts: conclusion-quality-eval-pipeline-prompts | Concept: sufficient-contextualization

Consolidates the redesign of the deriver-evaluation harness into three instruments with non-overlapping responsibilities, plus cross-cutting harness changes. Written to be handed to a coding agent; assumes the excadrill eval harness (evals/auxiliary/…) as the target.

The definition being implemented (from the sufficient-contextualization doc): a conclusion is good iff a future reasoning process can use it as a premise without seeing the source conversation (stranger test, sharpened to premise-usability), AND everything in it is licensed by what was said. Everything below operationalizes that one definition.

The three instruments and their separation of concerns:

InstrumentUnitMeasuresCost shape
Two-pass premise judge (NEW)per conclusioncontextualization quality + faithfulness2 calls / conclusion
Set-level bench (SLIMMED deriver bench)per batch/windowredundancy, temporal contradiction1 call / batch
Coverage bench (MODIFIED)per windowrecall, weighted by importancefew calls / window

No instrument duplicates another’s axis. Quality without coverage teaches the deriver to be conservative (mint 3 safe conclusions, skip everything hard); coverage without quality teaches it to spray. Both must run for hill-climbing to be safe.

Component 1: Two-pass premise-usability judge (NEW)

Prompts already written: premise_judge_prompts.py (PASS_1_BLIND_RECONSTRUCTION_PROMPT, PASS_2_SOURCE_VERIFICATION_PROMPT). This section specifies the harness around them. The molecular score is (decontextuality × minimality) ** 0.5 (molecular/__main__.py:214); ambiguity is a diagnostic feeder, NOT a scored multiplicand. The two-pass judge produces the decontextuality leg — replacing DECONTEXTUALITY_PROMPT and its AMBIGUITY_DETECTION_PROMPT feeder. MINIMALITY_PROMPT is retained as the other leg (separate axis), except: treat the 8–25 token range as a soft diagnostic, never a scored criterion — it penalizes legitimately long disambiguations.

1.1 Isolation requirement (correctness-critical)

  • Pass 1 MUST be a separate API call whose context contains ONLY the pass-1 prompt and the conclusion text. No source conversation, no sibling conclusions, no deriver prompt, no system context that reveals the source. If source text leaks into pass 1, the test degenerates into the self-report (“do you understand this?”) the design exists to avoid.
  • Add a harness assertion: the rendered pass-1 request must not contain any line of the window text (substring check against the logged window). Fail loudly, not silently.

1.2 Inputs

  • Pass 1: {conclusion}.
  • Pass 2: {conclusion}, {pass_1_json} (verbatim pass-1 output), {peer_id}, {window} = the exact message set (with timestamps) the deriver saw when this conclusion was minted. See Component 4.1 for window logging.

1.3 Verdict combination (harness code, not judge output)

def verdict(p1, p2):
    pass1_ok = (len(p1.unknown_slots) == 0
                and p1.licensed_inference != "NONE")
    pass2_ok = (p2.interpretation == "MATCH"
                and len(p2.unsupported_elements) == 0
                and p2.force_preserved
                and p2.time_status in {"ANCHORED_CORRECTLY", "SELF_ANCHORING",
                                       "HEDGED_APPROPRIATELY",
                                       "ANCHORED_TO_BAD_TIMESTAMP"})
    return pass1_ok and pass2_ok

Notes:

  • ANCHORED_TO_BAD_TIMESTAMP is a PASS for the deriver (it correctly trusted the timestamp it was handed; the timestamp was the failure). Count it separately as an input-quality metric — do not fold it into deriver failure rates.
  • OVER_INFERENCE in unsupported_elements fails the conclusion under current policy (deriver is explicit-only). Implement severity as harness policy, not judge behavior, so the explicit/deduced boundary can move without touching prompts. Config flag: over_inference_fails: bool = True.

1.4 Metrics to emit

Per run:

  • pass_rate
  • mean_decontextuality (pass-1 score; this is the gradient the old deriver bench lacked)
  • failure_attribution_split: counts of PRESENT_IN_WINDOW vs ABSENT_FROM_WINDOW vs ABSENT_EVERYWHERE across all failed conclusions. This is the headline RCA number: PRESENT_IN_WINDOW failures are fixable by prompt work (ML-332) now; ABSENT_FROM_WINDOW failures are unfixable by any prompt and require giving the deriver retrieval/prior-conclusion context at write time. Report it prominently.
  • label_histogram: free-text labels from both passes, clustered offline (exact-match count first; embedding clustering later if needed). Do NOT promote labels to an enum anywhere in code — store as strings.
  • time_status_histogram
  • bundled_rate (pass-1 bundled flag; cross-reference with minimality scores)

1.5 Judge pinning

Pin both passes to one fixed judge model; record model id + prompt hash in every result row. (Adavya’s coverage run was already flagged on judge-model choice; make provenance non-optional.) Reasoning/thinking mode on the judge: allowed, but if enabled it must be enabled for the entire calibration run and all subsequent runs — never toggled between.

1.6 Calibration gate (must pass before trusting run output)

  1. Run pass 1 alone on the specimen gallery from the contextualization doc (~15 pre-adjudicated cases, pass and fail tables). The judge must reproduce the adjudicated verdicts. Specifically: courtland responded with 'yes'… must score ≤0.4 with OBJECT_OR_CONTENT unknown; Bob, Alice's boss at ACME, is retiring in November 2026 must score ≥0.8; michael said that he could murder Dwight must have force preserved and be usable. Encode these as unit tests on judge output.
  2. Build the human gold set: 100–150 conclusions sampled from production + bench traces, labeled pass/fail independently by 2–3 people, disagreements adjudicated by discussion (log the adjudication rationale — it is rubric-sharpening material). Measure judge–human agreement overall AND per failure mode (a judge can hit 90% overall while catching 0% of missing-object cases; report per-label recall).
  3. Disagreement harvest (free calibration data): run old deriver bench and new judge on the same traces once; every conclusion the old bench passed and the new judge failed is either a validated catch or a new-judge false positive. Route these to the human labeling queue first — they are the most informative examples per unit of labeling effort.

Component 2: Deriver bench — strip per-observation, keep set-level

Target: evals/auxiliary/deriver/ (JUDGE_SYSTEM / JUDGE_USER / JUDGE_TOOL).

2.1 Delete

  • The entire PER-OBSERVATION FAILURES section (hallucination, over_inference, attribution_failure, misattribution, misframed, meta_conversational, tautology) and the corresponding observation_issues machinery in JUDGE_TOOL. All of it is subsumed by the two-pass judge (mapping: hallucination→FABRICATION, over_inference→OVER_INFERENCE, misattribution→license check, misframed→force preservation, tautology & meta_conversational→licensed-inference test). Running two per-observation judges with different quality definitions produces unactionable disagreement.
  • Both temporal NOTE paragraphs (“Temporal precision is handled by the storage system… do not flag observations for lacking time references” and “Missing temporal qualifiers are NOT an issue”). These encode a false premise: storage timestamps are batch-ingest time, not utterance time, and custom backdates exist precisely because the system cannot attach temporal context automatically. If any transitional period runs the old bench alongside the new judge, these paragraphs guarantee contradictory verdicts on every unanchored statement. Delete regardless of what else ships, ideally today.
  • TOPICAL_FIXATION: moves to coverage bench (it is a skew/recall property; see 3.5).

2.2 Keep (this becomes the whole bench)

Set-level, one call per observation batch with its window:

  • REDUNDANCY: near-duplicates and generation loops across the batch (the “EEO compliant 4× in six minutes” failure — invisible to any per-conclusion judge).
  • TEMPORAL_CONFUSION, narrowed to actual contradictions only (same person two places at once, “planning to go to X” + “is in X”). Missing anchors are the premise judge’s job.

Rename to set_level_bench (or similar) to stop implying it is the deriver’s quality definition. Output schema: keep the existing set_issues array + severity; drop valid_observation_numbers (no longer meaningful).

2.3 Salvage into other components

  • The subject-first checking procedure from JUDGE_USER (does it name a subject → is it the right speaker → did the fact come from that person’s statements) is a good ordered decision tree: fold it into pass 2’s license-check instructions as procedure.
  • The “faithful paraphrasing is NOT over-inference” calibration note: add an equivalent guard to pass 2 (prevents over-flagging).
  • The empirically derived subtypes (image_to_possession, meaning_inversion, biographical_fabrication, etc.): keep as (a) seed examples in the free-text label vocabulary documentation and (b) sampling strata for the human gold set. Do not reinstate as an enum.

Component 3: Coverage bench — keep, with five changes

Target: GOLD_EXTRACTION_PROMPT, COVERAGE_MATCHING_PROMPT, QA_GENERATION_PROMPT, QA_VERIFICATION_PROMPT (all in coverage/__main__.py). Coverage is structurally required: it is the counterweight that makes conservatism costly under the premise judge. Do not axe.

3.1 Importance-weighted coverage; TRIVIAL leaves the denominator

  • Score = importance-weighted fraction of gold facts COVERED (suggested weights: CRITICAL 4, IMPORTANT 2, MINOR 1, TRIVIAL 0).
  • TRIVIAL gold facts are excluded from the score entirely. Optionally report trivial_skipped_rate as a positive signal (the silence principle: a deriver declining to mint inert facts is behaving ideally, not missing coverage).
  • Apply the vacuity rule to gold itself: add to GOLD_EXTRACTION_PROMPT a final filter — for each candidate gold fact, state one non-vacuous inference it licenses; if none, drop it from gold. Gold facts that license nothing should not exist.

3.2 Split the metric on the inference flag

Rule 4 of gold extraction already marks requires_inference; currently ignored in aggregation. Emit two numbers:

  • coverage_explicit: gold facts with requires_inference = false. This is the deriver’s score (the deriver is instructed explicit-only; penalizing it for obeying that instruction is the instruction-conflict trap).
  • coverage_implicit: gold facts with requires_inference = true. This is the measurement of the not-yet-built “dreamer” stage — report it, do not hill-climb the deriver on it. It quantifies what the dreamer is worth before it is built.

3.3 Anchor gold to the same standard as conclusions

GOLD_EXTRACTION_PROMPT currently models relative time (“started current job within the past year”) — gold with a shelf life. Change the TEMPORAL category instruction + example: resolve relative references against message timestamps into absolute anchors; when the timestamp is untrustworthy or absent, hedge the anchor explicitly (same rule as the deriver: never mint a confident date without basis). Gold must pass the standard it grades against.

3.4 Make QA verification generative

QA_VERIFICATION_PROMPT currently asks “determine if the extracted facts provide enough information to answer” — the unfalsifiable self-report again. Restructure:

  • QA_GENERATION_PROMPT: emit question AND answer key derived from the source ({question, gold_answer, source_span}).
  • QA_VERIFICATION_PROMPT: judge sees ONLY the extracted facts + question (not the source, not the gold answer) and must WRITE the answer, or UNANSWERABLE. Same isolation rule as pass 1: assert no source text in the rendered request.
  • New comparison step (cheap call or harness string/entailment check): grade the written answer against the gold answer → ANSWERABLE_CORRECT / ANSWERABLE_WRONG / UNANSWERABLE. ANSWERABLE_WRONG is a distinct and important bucket: the extraction contains something on-topic but misleading — route these conclusions to the premise judge’s labeling queue.

QA is the independent measurement path for the whole pipeline: it does not inherit the gold extractor’s blind spots (gold-matching has an oracle problem — LLM gold judged by LLM; shared blind spots vanish from the metric). Weight it accordingly in dashboards.

3.5 Adopt topical fixation here

Moved from deriver bench: report topic distribution of extracted facts vs topic distribution of gold facts; flag when a topic exceeding N% of gold is entirely absent from extraction, or when one topic exceeds 60% of extraction against a diverse gold set. Implementation freedom on topic assignment (judge-labeled topics per fact is fine).

3.6 Composite

Wire the existing (currently unused) quality×coverage filter: per gold fact that is COVERED, look up the covering conclusion’s premise-judge verdict; a gold fact covered only by failing conclusions counts as PARTIAL at best. The two numbers must never be read separately — charitable entailment matching means a badly contextualized conclusion can still be COVERED (e.g. gold “User works at Google”, extracted “he works at Google”). The composite is what makes that charity safe.

3.7 Soundness ordering

This bench has the least earned trust in the pipeline (Adavya’s run: one dataset’s coverage output unusable, judge-model choice flagged; plus the gold-oracle circularity). ML-329-style instrument verification hits this bench FIRST. Cheapest verification: human-audit LLM gold sets on 5–10 windows — gold extraction is much easier than derivation (full window, no batching pressure), so auditing is fast, and every gold error found is simultaneously a bench correction and a labeled example for the judge calibration set (double duty with 1.6.2).

Component 4: Cross-cutting harness changes

4.1 Window logging at derivation time (prerequisite for everything)

At every deriver invocation, persist alongside each emitted conclusion: {window_message_ids, window_text_hash, message_timestamps, peer_id, deriver_prompt_hash, deriver_model_id, derived_at}. Pass 2’s failure attribution (1.4) is impossible without it. If existing traces (fp8 beam/longmem/locomo) lack exact windows, reconstruct best-effort from batching parameters and mark rows window_reconstructed = true.

4.2 Provenance on every result row

{judge_model_id, judge_prompt_hash, reasoning_enabled, bench_version, run_id}.

4.3 Production-shaped eval option

Config to mirror production batching (e.g. batch size 512, janitor period) so bench numbers and production numbers are comparable. Benchmark conversations are cleaner than production; expect the true numbers to be worse, and label runs dataset_shape: benchmark | production.

4.4 Sequencing (do not reorder 1→3)

  1. Freeze the current deriver prompt. Baseline before improvement, or the delta is unattributable.
  2. Implement Component 1 + 4.1; pass calibration gate 1.6.1 (specimen gallery unit tests).
  3. Baseline run on existing traces. Check predicted failure clusters — the current deriver prompt’s examples predict specific signatures: unanchored ages/states, over-inference labeled explicit (“lives in NYC”-type), dropped objects of speech-acts. If pass-2 labels cluster there, judge and diagnosis are cross-validated in one run.
  4. Components 2 and 3 changes; disagreement harvest (1.6.3); human gold set (1.6.2).
  5. Only then: deriver prompt rewrite (ML-332, two-stage enumerate-unresolved → add-only-what-resolves), measured against the frozen baseline.
  6. Use failure_attribution_split to decide the next investment: PRESENT_IN_WINDOW dominant → keep iterating prompts; ABSENT_FROM_WINDOW dominant → prioritize giving the deriver prior-conclusion/representation context at write time.

4.5 Non-goals (explicitly out of scope)

  • No new enums for failure types anywhere. Labels are free-text strings, clustered offline. The taxonomy is generated, never maintained.
  • No repair of corrupted input (speech-to-text garbles, wrong backdates). Detect and attribute (ANCHORED_TO_BAD_TIMESTAMP, garbage-entity labels) but do not attempt to fix — input quality is upstream of this pipeline.
  • Contradiction/collision machinery (why “Plastic Extrusions” was re-derived after its correction) is downstream of conclusion quality and not part of this spec.

Acceptance criteria

  1. Pass-1 isolation assertion exists and a test proves it fires on a seeded leak.
  2. Specimen-gallery unit tests pass (1.6.1), including the three named specimens.
  3. A full run on one existing trace set emits: pass_rate, mean_decontextuality, failure_attribution_split, label_histogram, time_status_histogram, coverage_explicit, coverage_implicit, QA answerable_correct rate, redundancy report — with provenance fields populated on every row.
  4. Old per-observation deriver-bench code is removed; set-level bench runs standalone at ≤1 call per batch.
  5. Gold extraction output contains zero relative-time facts on a spot-check of 3 windows.
  6. The quality×coverage composite is computed and present in the run report.

Graph View

Backlinks