RCC · per-turn natural dialogue · external corpus · audited in a separate model session operated by the author

Per-Turn Certified Skip

One 3B model in both arms, run turn by turn like a real assistant. Once the answer has stabilised, arm B stops calling the model for the rest of the conversation. Zero regressions across 6,273 paired turns, 19.3% fewer tokens, and a separate adversarial model audit session, operated by the author (not a third-party institution), that reproduced every number from raw bytes — verdict PASS.

Arm-path tokens

−19.3%

2,386,346 → 1,926,411 · −459,935

Regressions

across 6,273 paired turns

Net correct

+107

3,088 → 3,195 · 0 turns made worse

Pareto gate PASS · separate adversarial model audit session (author-operated): OVERALL PASS (A/B/C), 0 blocking findings

What was run

Corpus

ABCD — Action-Based Conversations Dataset (ASAPP + Columbia, NAACL-HLT 2021, MIT). Real human-to-human support chats.

Cohort

Fresh held-out set never used before: 9–12 customer-turn conversations, non-query flows. 614 conversations, 47 intent labels. Zero overlap with earlier runs (hash-verified).

Ladder

Every customer turn, k = 1…T — the real deployment shape. 6,273 paired probes.

Task

From the customer turns so far, name the conversation intent. Exact match vs the corpus label. No LLM judge.

Model

ministral-3b-2512, temperature 0, both arms.

Arm A

Plain baseline. Calls the model every turn.

Arm B

Same model + controller. Once two prior answers agree, match a prefix-only classifier, and its confidence clears a train-fitted threshold (k ≥ 5), it locks and carries — no more calls this conversation. Otherwise it emits arm A's bytes exactly.

Cost

6,273 calls, all arm A; arm B made none. $0.239.

Result by turn

Turn k	A correct	B correct	Skipped	Token saving
1	227	227	0	0.0%
2	299	299	0	0.0%
3	319	319	0	0.0%
4	317	317	0	0.0%
5	316	324	136	−22.3%
6	310	322	159	−26.0%
7	306	320	174	−28.4%
8	312	326	197	−32.1%
9	304	330	209	−34.0%
10	212	228	143	−34.3%
11	111	123	77	−33.3%
12	55	60	37	−36.3%

Turn count falls after k=9 because conversations end. Skipped = turns where arm B issued no call. The controller cannot fire before turn 5 by construction.

Token saving per turn. Flat at 0 while the model is still forming the answer (turns 1–4); it stops re-asking once the answer stabilises, and the saving climbs to a third of every deeper turn.

Separate adversarial model audit session, operated by the author — verdict PASS

The audit session (a separate model session run by the author, given only the frozen artefacts and told to trust nothing — not a third-party institution) recomputed all 18 hashes and every headline number from the raw bytes, read-only, with no network or secrets. Verdict A/B/C all PASS, OVERALL PASS, zero blocking findings.

The ratchet is structural: only two code paths set arm B's output — arm A's bytes, or arm B's own earlier certified answer. The audit session re-derived all 6,273 route receipts byte-for-byte; 0 violations on the 5,141 fallback and 1,132 certified turns.
Observed regressions were zero in this frozen cohort. The two-path output ratchet is structural, but zero regression is not a universal architectural guarantee: in this cohort the three wrong carries occurred only on conversations where arm A remained wrong at every later measured turn; the observation holds across confidence thresholds 0.95–0.99.
Gold-blindness is measured by a journaled file-open hook (0 gold-file opens), and the controller thresholds were fitted on the training split only — re-derived independently.
The pre-run estimate (66% certified, 38% saving) was optimistic versus the actual (35%, 19%) — so the numbers cannot have been tuned after the fact.

Read it honestly

Token saving is an accounting fact. 1,132 calls were not made. Routes are deterministic. But it assumes a workload that re-queries every customer turn with full context and no provider prefix-caching; a last-turn-only workload saves 0%, and with cached-input pricing the figure is ~12–18%.

The +107 is arm A's noise, absorbed — not the model getting smarter. After the answer stabilises, arm A still flips it ~8.5% of the time; arm B freezes the stable answer and avoids those flips. A perfectly consistent model would show +0. The honest claim is "In this frozen cohort, arm B was not worse than arm A and absorbed observed arm-A inconsistency", not "accuracy improved".

The controller fires on easy conversations. Arm A was already right 87% of the time on the conversations that got certified, vs 29% on the rest. That is correct behaviour for a conservative controller, but the savings and the +107 are earned on the easy third.

Same mechanism as before, more room to act. This is "the answer is settled, stop re-asking" — the same idea as the sparse-ladder run, now run per-turn so certification at turn 5 saves every later turn. It does not show the model itself narrowing at depth; the 3B model plateaus at ~49% while a bag-of-words classifier on the same text reaches ~85%.

Provenance

Result

authoritative_result.json
3b50a0d2…5c9b4

Journal

journal.jsonl · 18,826 lines
a0060852…89ca9c

Controller

controller.py
8c342739…caa490

Corpus file

abcd.json.gz
2bdf53ac…6de15

Params

controller_params.json
5119aca0…c983bfc

Ledger

6,273 calls · 0 retries · 0 arm-B calls

RCC multiturn benchmark · per-turn certified-skip (v3) · ministral-3b-2512, both arms · separate author-operated audit session: OVERALL PASS. Prior lanes: v1 sparse ladder (−7.5%, PASS), v2 compressed-prompt (−22% but 58 regressions, FAIL — asking the model a different question is not ratchet-safe with this model). v3 keeps the prompt identical and only decides whether to call.