Recovery Labels Map to a Skill. They Don't Yet Prove It's Taught.

The labels tell us why an example was collected, but they do not yet show whether fine-tuning taught the underlying skill. Before reading the coverage numbers, here is what would count as evidence that the model learned it, and where this post sits on that ladder. Recovery data is often organized under labels such as user correction, contradictory follow-up, wrong inherited context, partial result, tool failure, retry, and clarification. Those labels are useful for quota-setting and dataset balance, but they describe the trigger for recovery more clearly than the skill the agent needs to demonstrate.

What would count as evidence here

Here is the evidence ladder, from weakest to strongest.

Weak

• Match each eval case to a trigger and required action; score pass/fail against design gold.
• Treat this as target-match evidence, not a seen/unseen generalization test.

Better

• Split each family into seen-in-training and held-out cases, with n and pass rate for each scored cell.

Strong

• Pre-register valid×seen, valid×unseen, and invalid cases, and report enough n to support rates rather than anecdotes.

This post now meets the first tier for several families: eval cases scored pass or fail against design gold. It now partly meets the second: for the contradiction and unknown-term cases, each eval prompt can be labeled seen or unseen against the delivered training (worked pair below). The other families are not yet split.

Recovery as a state-handling claim, not just a trigger

Using the framework from the first post, each recovery trigger can be restated as a transition claim: what part of the state has to stay fixed, what part is allowed to change, and whether that specific change was demonstrated in training or has to be handled without having been seen. A user correction that changes one field, for example region, while every other field, business unit, metric, and fiscal period, stays exactly as it was, is a valid transition. Whether it counts as a training-exposure case or a held-out generalization check depends only on whether that specific field-level correction showed up in the authored conversations.

A few terms up front

A user correction may require changing one field while leaving several others untouched. A contradictory follow-up may require stopping to ask a clarifying question before altering state. A tool failure may require retrying execution while preserving the user's request exactly. Those are different skills even when they sit in the same high-level recovery set.

What the underlying numbers showed

In the analyzed multi-turn asset, about 80% of recovery conversations contained at least one visible state change, compared with just over 90% of standard conversations. The family-level split was more informative: user-correction conversations showed a visible state change around 95% of the time, wrong-inherited-context cases were in the low 90s, and contradictory follow-ups were around a quarter.

A low rate here isn't obviously a problem. If the correct response to a contradiction is to withhold any change and ask, most contradictory follow-ups should show no visible state change by design.  

The delivered conversations settle the call. All 25 contradictory-follow-up conversations contain a follow-up turn that makes no tool call, and in 20 of them that turn asks or flags the conflict in words (for example, "I see two different Business Units in the same follow-up. Which BU do you want me to carry forward?"). Eight of the 25 do show a state change, but only after the user answers. The low rate is therefore the skill working, and the success metric for this family should be withhold-and-ask rather than state change.

These figures describe training-set exposure, not evaluation performance. The percentages above come from a sample of roughly 1,400 multi-turn conversations, about 500 tagged as recovery and about 900 standard, which is the denominator behind the aggregate 80 percent versus 90 percent comparison. The delivered data registry supplies the per-family counts behind those shares (table below).

The original coding rule for “visible state change” is not documented, so the 95 percent, low-90s, and roughly 25 percent split should be read as approximate. The next section restates the measure under an explicit rule that anyone can rerun.

Per-Family counts under an explicit rule

Rule used: a conversation shows a visible state change when business unit, region, fiscal period, analytical dimension, metric, or operation differs between two consecutive tool-call queries. This is a change in the tool-call arguments, not a difference in the output. Counts come from the delivered 1,400-conversation multi-turn package and its registry, which is the 900 standard plus 500 recovery split behind the aggregate comparison above.

Training exposure only, replicated from the delivered package and registry. These rates say nothing about model performance.

Under this rule the aggregates land close to the earlier figures (standard 94.6% against just over 90%, recovery 82.0% against about 80%, contradictory 32.0% against about 25%), but user correction is 83.3% rather than about 95% and wrong inherited context is 87.7% rather than the low 90s. The earlier family-level split probably used a looser rule. Two labels from the trigger list above, tool failure and retry, are not among the nine delivered types. The only tool-error recovery units are 32 in the later 3,000-unit package, 0.4% of the 8,323 delivered units.

Evaluation-side scores now exist at the category level, though against the eval harness's own taxonomy rather than the recovery labels used here. They are reported, with the comparison model's results, in the section after the table below.

Add a required-action tag

A low-cost improvement is to retain the existing recovery trigger and add a second tag for the action the model is expected to take.

Why this matters: trigger tags alone can hide different skills. Without a required-action tag, you cannot tell whether you are under-covering “edit one field” or “ask before changing.” The table below is a starter map of those skills and the strength of the evidence for each, not a results table. All six worked examples below are grounded in delivered material: four include scored evaluation failures, one is an authored training conversation, and two reflect specified target behavior from the evaluation design.

First scored rows by required action

The table above carries a few scored failures but no pass rates. The client's V3 run (1 October 2026) now provides them, and a pass or fail from an LLM judge in a different model family is available for every eval case. Each row pairs a required action with the closest eval tag and with the number of delivered training units for the matching skill. The pairing is directional because the harness uses its own taxonomy.

Scored: client V3 run, judged pass or fail by an LLM from a different model family. Training units counted from the delivered packages.

The category figure can hide a weaker subset. Within the 62-record errors category (47 passed), the 25 grounding-after-empty cases passed 25 of 25 while the 25 tool-error recovery cases passed 11. Most of the 14 failures are not retry failures: 19 of the 25 cases use a business unit absent from the local CU-only data, and the system retried correctly but reported the empty result as a real $0. The eval therefore mixes two skills, retry and honest no-data reporting, and the 44% should not be read as a retry score.

What the scored eval adds

The scored evaluation now provides a related check on these behaviors, scored against its own eval taxonomy rather than the recovery labels used here, so treat the pairing below as directional, not a relabeling of the same categories. Across the full roughly 700-record evaluation set, the fine-tuned system passed 323 of 700 cases overall, compared with model 512 of 706 for a general-purpose comparison model (four further comparison-model runs errored).. Clarification and entity resolution, where an unrecognized name should trigger a question rather than a guess, passed 31 of 92 cases and relates to asking a clarifying question. Presentation-only follow-ups, which should reformat or re-sort a prior answer without re-querying, passed 20 of 60 and relates to editing one inherited field. Defaults and output discipline, which covers carrying forward the right default field rather than silently substituting one, passed 27 of 104 and relates to repairing wrongly inherited context. Errors, empty results, and tool failures, where the correct move is to retry or report rather than invent a number, passed 47 of 62 and relates to retrying a failed tool call.

The comparison model's scores on the same categories sharpen the picture. On defaults and output discipline it passed 100 of 104, against 27 for the fine-tuned system, and on presentation-only follow-ups it passed 41 of 60, against 20. Clarification is the exception: the comparison model passed only 38 of 92, close to the fine-tuned system's 31, which suggests that category is hard in its own right rather than a gap fine-tuning opened.

Those numbers are family-level pass rates, not seen-versus-unseen splits. They do not yet say which failures were training-exposure cases and which were held-out generalization checks. What they do say is that every one of these categories is a real, active failure point, not just a label a team assumes is well covered.

One scored follow-up makes the problem especially clear. After an earlier request established a top-five result, the user said, “narrow that to the top 3.” The expected behavior was to change only the active result count from five to three while preserving the rest of the request. The system kept five. The failure was therefore not a broad misunderstanding of the task. It was a failure to update one inherited parameter while leaving the rest of the active state intact.. The comparison model did narrow to three, but it switched from end customers to product end customers and matched only 33% of the expected rows, so the two systems failed this case for different reasons.

The goal is not to build a perfect taxonomy before the next authoring round. A small set of action tags usually emerges after reviewing a modest sample. The useful signal is whether a trigger label maps to one consistent skill or quietly contains several different recovery behaviors.

The eval design and the scored breakdown now largely overlap

The evaluation design and the scored breakdown now substantially overlap, though they are organized by different taxonomies: the eval harness scores against its own categories, not the recovery trigger labels used here, so the two should be read side by side rather than merged. The roughly 700-record evaluation set contains 204 multi-turn records, with explicit coverage for state retention, scope changes, clarification, and recovery, and the category-level scores above come from running the full set, multi-turn and single-turn alike. Fifty-five records are tagged for inherited context and explicit changes across turns. Twenty-five target multi-turn operation and scope inheritance. Another 25 cover tool-error recovery, with the same 25 tagged for corrected retry while preserving the requested scope. Ten test contradictions that should trigger clarification before a state change, and 23 test unresolved entity names that should be clarified before execution.

These categories overlap, so they should not be added together as a single recovery count. What is still missing is not the family-level score, which now exists at the eval-taxonomy level, but the seen-versus-unseen split within each family: how many scored cases were training-exposure and how many were held out, and whether the model performed differently across that line. That is the evidence needed before the family pass rates above can be read as a generalization claim rather than an overall capability check.

One real, delivered example is worth citing directly. When a user named two conflicting region values in a single request, the agent responded: “I see conflicting values in the requested scope... Which one should I use?” while explicitly holding the rest of the state unchanged. That confirms the “ask a clarifying question” row above describes a behavior that is actually demonstrated in delivered training data, not just a plausible skill on paper.

Recovery cases against the Blog 1 axes

Two cases fit directly onto the grid from the first post. The delivered clarifying-question example is a valid recovery behavior, but it sits outside the seen-or-unseen transition axis entirely: the correct move is to withhold the change and ask, so no transition is committed. The scored user-correction case in the table is a valid transition that the model failed, but nothing in this material says whether that specific metric correction was seen or unseen in training, so it cannot yet be placed in either cell.

The clarifying-question placement rests on an authored training example. The user-correction placement rests on a scored failure, but its training-exposure status is not established, so the seen-versus-unseen question for that case remains open.

A worked pair now fills the unseen cell. Valid and unseen, passed: none of the 10 contradiction eval prompts matches a training prompt or its masked template (41 training units), and the fine-tuned system passed all 10 (comparison model 0 of 10). Valid and unseen, failed: none of the 8 unknown-term prompts matches the 50 ambiguous-terminology units, and the fine-tuned system passed none (comparison model 4 of 8). For these two families the valid-and-seen cell is empty by construction, so every scored case is valid and unseen, and the same label supports opposite claims. That is why seen versus unseen has to be reported per skill. Template overlap here is a masked-text match, a conservative proxy for training exposure.

Turn count is a weak proxy for recovery difficulty

A ten-turn conversation can be easy if every turn follows the same pattern. A three-turn exchange can be harder if the final turn corrects information introduced earlier while several other fields must remain unchanged. For recovery, distance and dependency often tell more than raw conversation length. The review should capture where the relevant context was established, how far away the correction occurs, and what information must survive the repair.

Key takeaway: Tag recovery examples by both trigger and required action. That makes the recovery set useful for diagnosing which repair skills are well represented and which remain thin. Training percentages describe exposure, not generalization; they should not be read as a claim about how the model performs.