Premise Judge Prompts (supplementary)

Prompt constants for conclusion-quality-eval-pipeline Component 1 (the two-pass premise-usability judge). Destined for premise_judge_prompts.py in the excadrill harness. Verbatim below.

---

"""Two-pass premise-usability judge for conclusion quality.

Replaces the field-checking DECONTEXTUALITY_PROMPT with a generative test: pass 1 forces the judge to reconstruct the proposition blind and demonstrate premise-usability; pass 2 grades that reconstruction against the source window.

HARNESS WIRING (non-negotiable for the design to work):

1. Pass 1 is a SEPARATE API call. The judge must not have the source conversation, the deriver prompt, or any sibling conclusions in context. If the source leaks into pass 1, the test degenerates back into a self-report.
2. Pass 2 receives: the conclusion, the full pass-1 JSON output, and the ENTIRE input window the deriver saw when it minted this conclusion (all messages with their timestamps + target peer id). Log the window alongside each conclusion at derivation time; pass 2’s failure attribution depends on it.
3. Scope: this judge is per-conclusion. Coverage (did the deriver miss things) and redundancy (near-duplicates across the batch) are set-level properties over the window/batch and need separate instruments. Do not bolt them on here.

VERDICT COMBINATION (computed in harness code, not by either judge): PASS iff: - pass 1: no UNKNOWN slots (for slots the claim requires) AND licensed_inference != “NONE” - pass 2: interpretation == “MATCH” AND no unsupported elements AND force_preserved AND time_status in {“ANCHORED_CORRECTLY”, “HEDGED_APPROPRIATELY”, “SELF_ANCHORING”, “ANCHORED_TO_BAD_TIMESTAMP”} (the last one is an input failure, not a deriver failure) Otherwise FAIL, with primary_cause taken from pass 2’s attribution.

HEADLINE METRICS TO AGGREGATE:

• pass rate; mean decontextuality score (keeps a gradient the deriver bench lacks)
• the failure-attribution split: PRESENT_IN_WINDOW vs ABSENT_FROM_WINDOW. This is the single most decision-relevant number the bench emits: it says whether prompt work (ML-332) or input-context work is the binding constraint.
• free-text failure labels, clustered offline. These are the empirical, generated fill for the DeriverBench taxonomy sections. Do not promote them to an enum. """

PASS_1_BLIND_RECONSTRUCTION_PROMPT = """You are evaluating whether a stored conclusion can function as a PREMISE for future reasoning by a reader who will NEVER see the conversation it came from.

You are that reader. You have the statement below and nothing else. Do not imagine, guess, or reconstruct the missing conversation. Your reconstruction will later be graded against the real source: anything you invent will be caught, and inventing counts against you. When you cannot fill a slot from the statement’s own words, write UNKNOWN.

Task A: Reconstruct the proposition

Fill these slots using ONLY what the statement itself says:

• SUBJECT: Who or what is this claim about? Must be uniquely identifiable to a stranger. A bare first name is acceptable only if it is an established peer id convention; flag it if it could collide.
• PREDICATE_AND_FORCE: What did the subject do / say / believe / feel / plan / prefer? Preserve force markers exactly: “said”, “believes”, “might”, “did”, “was told”, “joked”. Note whether the claim is something the subject SAID versus something they DID — these license different inferences.
• OBJECT_OR_CONTENT: What is the act ABOUT? A reply requires what it replied to. An emotion requires its object. A plan requires its content. An agreement requires the proposal agreed to. A preference requires what is preferred.
• TIME: When does the claim hold? Enduring attributes (a birthday, a name, a portmanteau’s derivation) are SELF_ANCHORING. State claims (“is excited”, “works at”, “is 25”) are time-dependent and require an absolute anchor. Relative or missing anchors on time-dependent claims are UNKNOWN.
• SCOPE: If the claim is a preference, habit, or generalization: under what conditions does it hold? Write N/A for simple event claims.

The vacuity rule (critical)

A slot filled only by a bare existential does NOT count as filled. “A question”, “someone”, “something”, “a project”, “a woman” name THAT an object exists, not WHAT it is. Treat these as UNKNOWN. This applies even when the statement itself contains the vacuous phrase — “responded ‘yes’ to a question” has an UNKNOWN object, because no future reasoning can use “a question”.

Task B: Licensed inference

State ONE concrete inference or hypothesis a future reasoner could form using this statement as a premise, alone or combined with one plausible premise from a different conversation. The inference must depend on the SPECIFIC content of this statement.

Anti-vacuity check: if your best inference would remain equally true with the statement’s specifics replaced by “something / someone / at some point”, it is not an inference — write NONE. A premise that licenses nothing is inert regardless of how many slots are technically filled.

Task C: Decontextuality score

• 1.0: every required slot filled, non-vacuous inference produced
• 0.8: one minor gap (e.g. weakly disambiguated subject) but the proposition is complete and usable
• 0.6: usable core but a required anchor is missing (e.g. time-dependent claim with no date)
• 0.4: a load-bearing slot is UNKNOWN; the proposition is incomplete
• 0.2: multiple load-bearing slots UNKNOWN; the statement is parseable but inert
• 0.0: no recoverable proposition

Task D: Labels

Emit short free-text labels for each defect you found. Examples of the KIND of label (not an enum — coin new ones freely): missing-object-of-reply, unresolved-pronoun, relative-time, unanchored-state-claim, vague-referent, flattened-force, precision-without-context, bundled-claims, vacuous-object.

Also set BUNDLED: true if the statement carries more than one independently verifiable claim (count truth values, including subordinate clauses like “who owns a dog”). Bundling is scored elsewhere; here you only flag it.

Worked examples

Statement: “courtland responded with ‘yes’ on July 2, 2026 at 2:13:45 AM” → SUBJECT: courtland. PREDICATE_AND_FORCE: said (verbatim reply). OBJECT_OR_CONTENT: UNKNOWN — the thing assented to is absent; “yes” alone is not content. TIME: July 2, 2026. SCOPE: N/A. → LICENSED_INFERENCE: NONE — “courtland agreed to something at some point” survives the specifics being deleted. → SCORE: 0.2. LABELS: [missing-object-of-reply, precision-without-context]. BUNDLED: false.

Statement: “Bob, Alice’s boss at ACME, is retiring in November 2026” → SUBJECT: Bob, uniquely resolved via enduring descriptors (Alice’s boss at ACME). PREDICATE_AND_FORCE: is retiring (stated as fact). OBJECT_OR_CONTENT: retirement from the ACME role. TIME: November 2026. SCOPE: N/A. → LICENSED_INFERENCE: “A management opening at ACME after November 2026 could be Bob’s former position.” → SCORE: 1.0. LABELS: []. BUNDLED: false.

Statement: “michael said that he could murder Dwight” → SUBJECT: michael. PREDICATE_AND_FORCE: said + could — force preserved; reads as hyperbole, not assertion of intent. OBJECT_OR_CONTENT: hyperbolic statement about Dwight. TIME: UNKNOWN (event claim, no anchor). SCOPE: N/A. → LICENSED_INFERENCE: “Michael was frustrated with or performing exasperation about Dwight at the time of utterance.” → SCORE: 0.6 (usable, but the utterance is unanchored). LABELS: [relative-time]. BUNDLED: false.

Statement: “alice is 25 years old” → TIME: UNKNOWN — an age is a time-dependent state; true for one year, then silently false. The self-anchoring form is a birth date or “turned 25 on “. → SCORE: 0.6. LABELS: [unanchored-state-claim].

Output format

Return ONLY a JSON object: { “subject”: ”…”, “predicate_and_force”: ”…”, “object_or_content”: ”…”, “time”: ”…”, “scope”: ”…”, “unknown_slots”: [”…”], “licensed_inference”: ”… or NONE”, “decontextuality_score”: 0.0, “labels”: [”…”], “bundled”: false, “notes”: “one or two sentences, only if something needs explaining” }

Statement to evaluate: {conclusion} """

PASS_2_SOURCE_VERIFICATION_PROMPT = """You are grading a blind reading of a stored conclusion against the source it was derived from.

A first judge read the conclusion WITHOUT the source and reconstructed the proposition it carries (JSON below). You have what that judge lacked: the full input window the deriver saw. Your job is to determine (1) whether the blind reading was correct, (2) whether the conclusion is licensed by the source, and (3) for every gap the blind judge found, WHERE the failure lives — because “the deriver dropped available context” and “the context was never in the window” demand entirely different fixes.

Question 1: Interpretation match

Compare the blind reconstruction to what the source actually shows the statement meant.

• MATCH: the blind reading (including its UNKNOWNs) is consistent with the source.
• MISMATCH: the blind judge confidently reconstructed the WRONG proposition — wrong referent, wrong object, wrong force. This is the worst failure class: the conclusion doesn’t just underdetermine its meaning, it actively misleads. Typical cause: context inherited across a topic switch.
• N/A: the blind reading was mostly UNKNOWNs; there is nothing substantive to match.

Question 2: License check

For EVERY element of the conclusion — subject, predicate, object, time, scope, and every subordinate descriptor riding along (e.g. “his dog Lex”) — is it supported by the window? Classify each unsupported element:

• OVER_INFERENCE: a plausible leap presented as stated fact (source: “took my dog for a walk in NYC” → conclusion: “lives in NYC”).
• FABRICATION: no basis in the window at all.
• MISATTRIBUTION: stated by or about someone other than the conclusion’s subject; includes the assistant’s own suggestion reflected back as the user’s claim.
• STALE_CONTEXT: real content from the window attached to the wrong statement (the switch-point failure).

Note: a legitimate deduction may be a fine conclusion — but if the deriver’s task was explicit extraction, label deductions as OVER_INFERENCE and let the harness decide severity. The license check protects the boundary; it does not decide policy.

Question 3: Force preservation

Does the conclusion’s force match the source’s? Check: hedges kept (“might”, “could”), said-vs-did preserved, attribution kept (“was told that” vs “believes”), register kept (a joke stored as a joke, not as a belief). FORCE_PRESERVED: true/false, with the specific flattening named if false.

Question 4: Time provenance

How was time handled, judged against the message timestamps in the window?

• ANCHORED_CORRECTLY: relative references resolved against the message’s stated timestamp.
• SELF_ANCHORING: no time needed (enduring attribute).
• HEDGED_APPROPRIATELY: anchor was genuinely untrustworthy or absent, and the conclusion preserved the uncertainty (e.g. “alice said, in a message dated July 10, 2026, that she went ‘yesterday’”) instead of guessing.
• ANCHOR_MISSING: relative or absent time on a time-dependent claim.
• ANCHOR_GUESSED: an absolute date was minted with no trustworthy basis.
• ANCHORED_TO_BAD_TIMESTAMP: the deriver correctly trusted the timestamp it was handed, and the timestamp itself was wrong (bad backdate, batch-ingest time). This is an INPUT failure, not a deriver failure — say so.

Question 5: Failure attribution (the decision-relevant one)

For EACH slot the blind judge marked UNKNOWN, and each defect found above:

• PRESENT_IN_WINDOW: the resolving context was in the deriver’s input and the deriver dropped it. → prompt/model failure; fixable by ML-332-style prompt work now.
• ABSENT_FROM_WINDOW: the resolving context existed in the conversation’s past but had left the token-capped window before derivation. → architecture/input failure; no prompt can fix it; needs context-retrieval-at-write-time.
• ABSENT_EVERYWHERE: the utterance never contained the resolving information (speaker said “that thing” and never disambiguated). → the correct deriver output may have been a hedged statement or silence; judge whether what was minted was better than nothing.

Output format

Return ONLY a JSON object: { “interpretation”: “MATCH | MISMATCH | N/A”, “interpretation_notes”: ”…”, “unsupported_elements”: [ {“element”: ”…”, “classification”: “OVER_INFERENCE | FABRICATION | MISATTRIBUTION | STALE_CONTEXT”, “evidence”: “brief pointer into the window”} ], “force_preserved”: true, “force_notes”: ”…”, “time_status”: “ANCHORED_CORRECTLY | SELF_ANCHORING | HEDGED_APPROPRIATELY | ANCHOR_MISSING | ANCHOR_GUESSED | ANCHORED_TO_BAD_TIMESTAMP”, “failure_attribution”: [ {“gap”: “which slot or defect”, “attribution”: “PRESENT_IN_WINDOW | ABSENT_FROM_WINDOW | ABSENT_EVERYWHERE”, “evidence”: “where in the window the context is, or why it is absent”} ], “labels”: [“free-text failure labels; coin new ones freely”], “primary_cause”: “one label naming the dominant failure, or NONE if clean” }

Conclusion under evaluation: {conclusion}

Blind reconstruction (pass 1 output): {pass_1_json}

Target peer: {peer_id}

Full input window the deriver saw (messages with timestamps): {window} """