Falinks — Human Labeling for Judge Calibration

Repo plastic-labs/falinks

Falinks collects human labels for calibrating the Excadrill action judge, which evaluates conclusions produced by the deriver on eight criteria. The Falinks web app provides 1-2 labeling pages for five of those criteria and exports the resolved human verdicts for Excadrill’s calibration report.

Three excluded criteria

The Deriver action judge scores conclusions across 8 axes — decontextuality, entailment, atomicity, triviality, temporal, economy, coverage, and redundancy. Falinks samples and labels the first five. Excadrill evaluates the remaining three criteria through separate procedures (Falinks defines no corresponding tasks):

• Economy Excadrill removes each of a conclusion’s non-core segment in turn and reruns decontextuality, entailment, and temporal on the ablated conclusion. Economy is defined as the fraction of tested segments for which removal changes at least one of the three verdicts. Falinks already calibrates all three verdicts against human labels, so a separate economy label would repeat the same evaluation.

• Coverage Excadrill treats each explicit, verifiable fact in the conversation as required content. Coverage measures the fraction of those facts stated or entailed by the conclusions. Routine labeling would require reading the entire conversation and reconstructing every required element. Human review instead spot-checks generated answer keys

• Redundancy Excadrill identifies duplicate conclusion pairs and topic clusters within one conclusion set. Redundancy measures both the duplicate rate (conclusions belonging to at least one duplicate pair) and fixation (the largest topic cluster as a fraction of the conclusion set). This evaluation compares conclusions within a set for duplication or near-duplication. A per-conclusion label cannot represent that comparison.

Calibration scores

Calibration compares each judge verdict with the resolved human verdict for the same conclusion. For labeled conclusion $i$, let $J_i\in \{\text{pass},\text{fail}\}$ denote the judge verdict and $H_i\in \{\text{pass},\text{fail}\}$ the resolved human verdict. Define the false-pass indicator ${\mathrm{FP}}_i$ and false-fail indicator ${\mathrm{FF}}_i$ as:

$$\begin{array}{rl}{\mathrm{FP}}_i& =1[J_i=\text{pass}\wedge H_i=\text{fail}],\\ {\mathrm{FF}}_i& =1[J_i=\text{fail}\wedge H_i=\text{pass}].\end{array}$$

Excadrill computes the calibration rates as:

$$\begin{array}{rl}\text{false-pass rate}& =\frac{{\sum }_{i:J_i=\text{pass}}{w}_i{\mathrm{FP}}_i}{{\sum }_{i:J_i=\text{pass}}{w}_i},\\ \text{false-fail rate}& =\frac{{\sum }_{i:J_i=\text{fail}}{w}_i{\mathrm{FF}}_i}{{\sum }_{i:J_i=\text{fail}}{w}_i}.\end{array}$$

where $w_i$ weights the calibration rates. Let $P_i$ denote the number of conclusions in the set before sampling and $m_i$ the number sampled from that set. Excadrill calculates:

$$w_i=\frac{P_i}{m_i}.$$

The $m_i$ sampled conclusions therefore contribute total weight $P_i$, equal to the number of conclusions in the set before sampling.

Excadrill reports each rate with a 95% Wilson interval evaluated at the effective sample size of its denominator:

$$n_{\text{eff}}=\frac{{\left({\sum }_i{w}_i\right)}^2}{{\sum }_i{w}_i^2}.$$

For each rate, $i$ ranges only over the conclusions in its denominator (i.e., judge-pass conclusions for false-pass rate and judge-fail conclusions for false-fail rate). The Wilson intervals do not adjust for stratified sampling or for sampling without replacement, so Excadrill reports them as approximate.

Human labeling

Falinks resolves each of its scored fields to its respective unique mode and then applies the criterion’s pass condition.

Decontextuality (2 pages)

Decontextuality measures whether a reader — given only the conclusion — recovers the intended core referents, claim, and scope established by the source conversation.

Page 1 · Blind interpretation. A labeler sees only the conclusion and restates it as specifically as the conclusion allows, marking each guessed part [GUESS]. The labeler also lists every reference the conclusion leaves unresolved — a pronoun, a name the conclusion does not identify, a pointer such as “that project,” or an unanchored time such as “last spring” — in the form term — what's missing. Falinks creates one Page 2 task for each submission.

Page 2 · Adjudication. A different labeler sees the conclusion, the blind interpretation, the reported unresolved references, and the source conversation, and assigns one of four verdicts:

Verdict	Assigned when
match	The interpretation expresses the same specific claim established by the source conversation.
wrong-referent	An entity or reference resolves to the wrong target.
wrong-claim	The interpretation resolves the referents correctly but asserts a fact that differs from the source conversation.
indeterminate	The interpretation commits to no definite reading, or the source conversation leaves multiple readings possible.

Let $v_{i,k}$ denote the verdict assigned to the blind interpretation written by labeler $k$ for conclusion $i$:

$$v_{i,k}\in \{\text{match},\text{wrong-referent},\text{wrong-claim},\text{indeterminate}\}$$

Falinks obtains the resolved verdict ${\hat{v}}_i$ by taking the mode of the verdicts on conclusion $i$ after adjudication of the required number of blind interpretations. A tie leaves no unique mode, so Falinks routes one further blind item, excluding every previous blind labeler and all labelers already exposed to the source conversation. Falinks routes the new response through the same Page 2 adjudication. Rounds continue until the mode becomes unique or no labeler remains eligible.

Falinks assigns $H_i=\text{pass}$ only when the resolved adjudication verdict equals match:

$$H_i=\text{pass}⟺{\hat{v}}_i=\text{match}$$

The judge runs the same two-stage protocol, replacing the two labelers with two calls to a single model. For the first call, Excadrill supplies only the conclusion. The model returns a restatement, the subject key to which the conclusion belongs, the unresolved references, and whether any of its [GUESS] markers affects the core referent or core claim. Before the second call, Excadrill removes an unresolved reference when a case-insensitive exact comparison identifies it as the subject key or its possessive form because the subject key identifies the peer the record describes, thus it is resolved by construction.

For the second call, Excadrill supplies the conclusion, restatement, remaining unresolved references, core-guess flag, and source conversation. The model returns match, wrong_referent, wrong_claim, or indeterminate. An unresolved reference or [GUESS] forces indeterminate only when it affects the core referent or core claim. Excadrill sets correct to true for match; for the other three verdicts, it sets correct to false and records the verdict in failure_mode. The judge verdict satisfies $J_i=\text{pass}$ exactly when the adjudication verdict equals match.

Diagnostics. A $4\times 4$ confusion matrix compares ${\hat{v}}_i$ with the judge’s verdict. A true correct value maps the judge verdict to match; a false value maps it to failure_mode. The report includes raw and weighted pooled matrices and raw and weighted matrices for each stratum. Within one batch, every conclusion in a stratum has the same sampling weight. A report that combines batches may contain different weights for the same stratum.

Temporal (2 pages)

Temporal validity measures whether a conclusion requires a time anchor and, when it does, whether the conclusion and available message timestamps determine an anchor supported by the dated source conversation.

Page 1 · Blind anchor. A labeler sees the conclusion and the timestamps of the messages cited in source_indices, without the messages themselves. The labeler answers whether interpreting the conclusion requires the time at which the asserted fact held. Answering no sets the status to not_required. Answering yes requires the labeler either to state a time or to mark cannot_tell. Marking cannot_tell sets the status to missing. Stating a time leaves the status unset and creates one Page 2 task. Falinks creates Page 2 only after an affirmative Page 1 response that states a time.

Page 2 · Adjudication. A different labeler sees the conclusion, the stated time, and the dated source conversation, then returns one of four verdicts which are mapped to a status:

Verdict	Status	Returned when
match	adequate	The dated conversation supports the stated time.
wrong-time	incorrect	The dated conversation contradicts the stated time or proposed time uses a granularity too broad to preserve the conclusion’s meaning.
indeterminate	missing	The dated conversation does not settle the stated time.
not-time-dependent	not_required	The conclusion needs no time at all, contradicting Page 1. Falinks records a disagreement flag alongside the status.

Let $t_{i,k}$ denote the status of the Page 1 answer submitted by labeler $k$ for conclusion $i$, whether that status came from the answer itself or from an adjudicator’s verdict on it:

$$t_{i,k}\in \{\text{not\_required},\text{adequate},\text{missing},\text{incorrect}\}$$

Falinks obtains the resolved status ${\hat{t}}_i$ by taking the mode of the statuses on conclusion $i$, with ties resolved by the decontextuality routing rule. Specifically, Falinks collects another Page 1 response. A response with status not_required or missing enters the mode calculation directly. A response that states a time proceeds to Page 2, which assigns the status used in the mode calculation. A cannot_tell response contributes missing directly and creates no Page 2 task because it contains no proposed time for adjudication.

A resolved status of not_required or adequate satisfies temporal validity:

$$H_i=\text{pass}⟺{\hat{t}}_i\in \{\text{not\_required},\text{adequate}\}$$

As with decontextuality, Excadrill uses one model call for the blind temporal assessment and a second call for adjudication. In the first call, Excadrill supplies the conclusion and the same timestamps shown on Page 1. The model classifies temporal dependence and returns either a date, a date range, or the string cannot determine when time-dependent. Neither answer sets a status.

For the second call, Excadrill supplies the conclusion, the first call’s time-dependence decision and proposed time anchor, the same timestamps, and the dated source conversation. The model returns one of the four adjudication verdicts (match, wrong-time, indeterminate, not-time-dependent), which Excadrill maps to a status. Excadrill assigns $J_i=\text{pass}$ only when the resulting status equals not_required or adequate.

Note that the human and judge routes use the same four statuses and pass condition but assign the statuses through different routes. Answering no to whether the conclusion requires a time anchor assigns not_required directly. A cannot_tell response assigns missing directly because it proposes no time anchor for Page 2 to assess. Excadrill, by contrast, sends every first-call response to the second call before assigning a status.

The human Page 2 task and the judge’s second model call both use the dated source conversation to reconsider the initial time-dependence decision. In both routes, a not-time-dependent adjudication replaces the initial time-dependent decision with not_required and records a disagreement. Both routes permit adjudication to override the initial time-dependence decision because the conclusion alone may not determine whether a time anchor is required. For example, User prefers the Berlin office suggests an ongoing preference, but the source conversation may restrict the claim to a decision about one past trip.

Evidence tiers. Temporal evaluation uses different timestamp evidence for cited and uncited conclusions. Excadrill assigns primary to cited conclusions and coarse to uncited conclusions. The primary tier supplies the cited-message timestamps to Page 1 and the first model call. The coarse tier supplies every message timestamp in the conversation. Excadrill reports the two tiers separately and uses the primary rates as the headline temporal results. The sampling report records eligible and selected counts for both tiers.

Diagnostics. A $4\times 4$ confusion matrix compares the judge status with the resolved human status. Cohen’s $\kappa$ compares their binary time-dependence values. Excadrill maps not_required to false and the other three statuses to true. When Page 2 returns not-time-dependent, this comparison uses the resulting not_required status rather than the affirmative Page 1 response (i.e., the Page 1 decision that a time anchor is required). The report separately counts these human overrides and disagreements between the judge’s first and second calls. It also lists every conclusion with a resolved human status other than not_required for which the resolved human anchor differs from the judge’s proposed anchor. These diagnostics do not affect the calibration rates.

Atomicity (1 page)

Atomicity measures whether a conclusion contains exactly one core claim.

Page · Atomicity. Labeler $k$ sees only conclusion $i$ and enters its number of distinct core claims, denoted by $c_{i,k}$. Falinks resolves the submitted counts by mode, yielding ${\hat{c}}_i$. A resolved count of one passes:

$$H_i=\text{pass}⟺{\hat{c}}_i=1$$

Excadrill’s LLM judge receives only the conclusion which it divides into independently verifiable propositions. Let $p_i$ denote the number of propositions. Excadrill applies the same pass condition:

$$J_i=\text{pass}⟺p_i=1$$

Diagnostics. Excadrill’s calibration report compares ${\hat{c}}_i$ with $p_i$ using exact-count agreement and mean absolute error. It reports both unweighted and sampling-weighted values.

Entailment (2 pages)

Entailment measures whether every element of a conclusion follows from the deriver’s cited evidence.

Page 1 · Claim split. A labeler sees only the conclusion and divides it into independently checkable claims.

Page 2 · Verdict. A different labeler sees the deriver’s cited evidence span and the claims identified by Excadrill’s LLM judge. The labeler assigns one of three statuses to each claim:

Status	Assigned when
supported	The cited evidence supports the complete claim.
unsupported	The cited evidence provides no textual basis for the claim.
over-inferred	The cited evidence supports part of the claim but not the complete claim.

For claim $e$ of conclusion $i$, let $v_{i,k,e}$ denote the status assigned by labeler $k$:

$$v_{i,k,e}\in \{\text{supported},\text{unsupported},\text{over-inferred}\}$$

Falinks resolves the status of each claim independently by mode. Let ${\hat{v}}_{i,e}$ denote the resolved human status for claim $e$ of conclusion $i$. The human verdict passes only when every claim has status supported:

$$H_i=\text{pass}⟺{\hat{v}}_{i,e}=\text{supported}\text{for every }e$$

Excadrill’s LLM judge independently receives the conclusion and the same cited evidence span shown on Page 2. In one call, the judge divides the conclusion into independently checkable claims and assigns one of the same three statuses to each claim. Let $v_{i,e}^J$ denote the judge status for claim $e$ of conclusion $i$. The judge verdict passes only when every claim has status supported:

$$J_i=\text{pass}⟺v_{i,e}^J=\text{supported}\text{for every }e$$

Diagnostics. A claim-level $3\times 3$ confusion matrix compares ${\hat{v}}_{i,e}$ with $v_{i,e}^J$. Claim-level false-pass and false-fail rates compare supported with the two non-supported statuses. Excadrill also records the absolute difference between the human and judge supported fractions for each conclusion. Finally, it compares the modal Page 1 claim count with the judge claim count. It reports unweighted and weighted match rates, signed count differences, and count mismatches. Falinks preserves every submitted claim list for manual inspection as equal counts may represent different claim boundaries.

Triviality (2 pages)

Triviality measures whether a conclusion could have been derived from most other conversations.

Page 1 · Informativeness. A labeler sees only the conclusion and marks whether it conveys information about the user (yes or no).

Page 2 · Execution. A different labeler sees the conclusion and reviews five comparison conversations one at a time. For each conversation, the labeler marks whether the conclusion could have been derived from it. Let $b_{i,k,j}$ denote the response from labeler $k$ for conclusion $i$ and comparison conversation $j$:

$$b_{i,k,j}\in \{\text{yes},\text{no}\}$$

Let ${\hat{b}}_{i,j}$ denote the mode-resolved human response for conclusion $i$ and comparison conversation $j$. A conclusion passes when no more than two comparison conversations permit derivation:

$$H_i=\text{pass}⟺\sum _{j=1}^{5}1[{\hat{b}}_{i,j}=\text{yes}]\le 2$$

Excadrill’s LLM judge makes one call for each of the same five comparison conversations. Each call receives the conclusion and one comparison conversation and determines whether the conclusion could have been derived from that conversation. Let $b_{i,j}^J$ denote the judge response. Excadrill applies the same pass condition:

$$J_i=\text{pass}⟺\sum _{j=1}^{5}1[b_{i,j}^J=\text{yes}]\le 2$$

Diagnostics. Excadrill reports unweighted and sampling-weighted disagreement rates between ${\hat{b}}_{i,j}$ and $b_{i,j}^J$. Cohen’s $\kappa$ compares the resolved Page 1 response with the judge’s triviality classification. A Page 1 response of no corresponds to a judge classification of trivial, while yes corresponds to non-trivial.

Sampling and holdout

Falinks samples judge passes and judge failures separately. Excadrill uses the judge-pass sample to estimate the false-pass rate and the judge-fail sample to estimate the false-fail rate.

Let $n$ denote the sampling target for each judge verdict. For decontextuality, entailment, atomicity, and triviality, Falinks targets $n$ judge-pass conclusions and $n$ judge-fail conclusions.

Triviality further divides each pass/fail target by proximity to the decision threshold. A judge-pass conclusion counts as boundary when exactly two of its five comparisons permit derivation. A judge-fail conclusion counts as boundary when exactly three comparisons permit derivation. Every other triviality conclusion counts as non-boundary. Falinks divides each $n$-conclusion target as evenly as possible between boundary and non-boundary conclusions.

Criterion	Conclusions eligible for sampling	Division before sampling
Decontextuality	Every imported conclusion.	Judge match conclusions and conclusions with any other judge verdict.
Entailment	Conclusions with explicit source-message citations and at least one judge claim.	Conclusions with every judge claim supported and conclusions with at least one non-supported judge claim.
Atomicity	Every imported conclusion.	Conclusions with exactly one judge proposition and conclusions with more than one judge proposition.
Triviality	Conclusions with a valid judge response for all five comparison conversations.	Judge-pass and judge-fail conclusions, each divided into boundary and non-boundary conclusions.
Temporal	Conclusions with at least one message timestamp and no unparseable timestamp.	One set for each judge status, namely not_required, adequate, missing, and incorrect.

For even $n$, triviality assigns $n/2$ conclusions to each boundary stratum and $n/2$ to each non-boundary stratum. For odd $n$, the passing boundary stratum and failing non-boundary stratum each receive the additional target slot.

Falinks targets $2n$ temporal conclusions. It distributes this target among the four temporal statuses so that the targets differ by at most one. When $n$ is odd, not_required and adequate each receive one more target slot than missing and incorrect.

When a stratum contains fewer eligible conclusions than its target, Falinks selects every eligible conclusion and records the difference as a shortfall. It does not reassign the unused target to another stratum.

Within each stratum, Falinks assigns approximately 20% of the eligible population and approximately 20% of the sampling target to the holdout partition. For either total $x$, the holdout count satisfies:

$$h(x)=\lfloor 0.2x+0.5\rfloor .$$

The remaining $x-h(x)$ conclusions belong to the non-holdout partition. Falinks samples the two partitions separately, so each partition has its own population size, sample size, and sampling weight.

Both partitions receive human labels. Only non-holdout labels may be used during prompt revision. Excadrill reports every calibration metric separately for the two partitions and applies an exact McNemar test within each partition when comparing two judge runs. Falinks does not record whether an operator used a particular label during prompt revision.

Batch construction

A batch collects the human labels for one sampling plan applied to one imported dataset. The batch request specifies an imported dataset, a sampling seed, and a target $n$. It also requires each initial task to be completed by one, two, or three different labelers. Falinks accepts the request only when the active roster contains at least twice the required response count, allowing the two entailment tasks and the two triviality tasks to use disjoint labeler sets.

Each selected conclusion-criterion pair creates the following initial tasks.

Criterion	Initial tasks
Decontextuality	One blind-interpretation task.
Entailment	One claim-split task and one verdict task over the judge-identified claims.
Atomicity	One core-claim-count task.
Triviality	One informativeness task and one five-conversation execution task.
Temporal	One blind-anchor task.

Falinks creates adjudication and tiebreak tasks only when required by the submitted responses. When the exclusion and source-exposure rules leave no eligible labeler for a generated task, Falinks records a routing shortfall and leaves the criterion unresolved. It retries routing shortfalls during database initialization and before each label submission.

A completed export requires every sampled criterion to resolve. A partial export omits unresolved criteria and recalculates the sample size and sampling weight for each affected criterion, sampling stratum, and holdout partition.

Judge-independent metrics for writing quality

Calibration determines how closely judge verdicts match resolved human verdicts. The resolved human verdicts also estimate how often deriver conclusions fail each criterion, providing the human reference for judge-based evaluation at scale. Because Falinks samples judge-result strata at different rates, Excadrill calculates a sampling-weighted human failure rate:

$$\bar{q}=\frac{{\sum }_i{w}_{i}1[H_i=\text{fail}]}{{\sum }_i{w}_i}.$$

Inter-labeler agreement

Judge–human disagreement does not establish judge error when human labelers also disagree with one another. Falinks calculates agreement separately for each type of human judgment. Tiebreak responses are excluded because Falinks requests them only after the initial responses disagree.

Falinks uses Fleiss’ $\kappa$ when every judgment in the calculation has the same number of labels and nominal Krippendorff’s $\alpha$ otherwise. Falinks reports pending until labeling is complete and the agreement statistic can be calculated. When both conditions are met, agreement of at least $0.7$ yields pass; lower agreement yields fix the guide and relabel. This status does not block export.

Serving order

Falinks serves blind tasks before source-revealing tasks to prevent source material from influencing blind judgments. The queue selects each labeler’s next task from the earliest eligible group in this sequence:

1. Decontextuality blind-interpretation and temporal blind-anchor tasks.
2. Conclusion-only atomicity, entailment claim-split, and triviality informativeness tasks.
3. Source-revealing decontextuality adjudication, temporal adjudication, entailment verdict, and triviality execution tasks.

After a source-revealing task exposes a labeler to a conversation, Falinks no longer assigns that labeler a blind task for a conclusion derived from that conversation. Falinks also excludes the author of a blind response from adjudicating that response.