The dataset had every value. The more useful question was what happened between them.

This series comes from a data audit of a multi-turn fine-tuning corpus containing several thousand delivered training units plus a separate evaluation set. The audit tracked state across tool-grounded turns using fields such as organizational unit, region, fiscal period, analytical dimension, metric family, and operation.

The transition concept itself is familiar. A stateful product should already have some definition of which values can change, which combinations are valid, and which transitions matter in normal use. The useful part of the audit was comparing that intended behavior with what the authored conversations actually demonstrated.

The training data was uneven across state fields

The full range of fiscal-period values appeared somewhere in the training data. When the conversations were examined turn by turn, however, zero atomic fiscal-period changes were observed across several thousand eligible successive state pairs. Other fields looked very different: organizational unit, metric family, and operation reached complete directed transition coverage among the observed values. Region reached about two-thirds, analytical dimension about 40%, and fiscal period remained at zero.

Without a specification, a coverage percentage is uninterpretable: a missing edge may be invalid, deliberately held out for generalization, or intended but absent from training. Each tier therefore implies a different evaluation job: high coverage tests uptake, partial coverage tests held-out generalization, and zero coverage calls for a probe of whether the behavior was ever specified.

The specification should come first

A data-derived transition list cannot tell a team about a valid transition that never appears anywhere in the corpus. If the transition is absent, it leaves no row to count. That is why the expected transition space should be defined independently from the training data, using the product, tool contract, or task specification.

Once that list exists, the audit becomes straightforward: identify the valid transitions, mark which are intended for training, mark any deliberate holdouts, and compare those sets with what was actually authored.

Coverage describes exposure; the scored example below asks what to evaluate when a valid transition is unseen.

An example from the evaluation set

One evaluation case answers this directly where a user asked for a metric broken out across two different quarters in a single turn. The system prompt tells the model to treat an unspecified period as a fixed default quarter, and the correct move was to override that default and report the two quarters the user actually named. Instead, since there was no earlier turn for it to fall back on, the model just used its own hardcoded default quarter and answered as if that had been the request, dropping both quarters the user asked for.

A human reviewer scored this response a 2 out of 7, noting that it doesn't account for multiple quarters and never returns a total for each one requested. The same thing happened again in an unrelated case asking for a comparison across all four quarters of the year, where the model once more ignored the requested range and returned a result for its default quarter alone, producing a variance figure that was meaningless since it was comparing that quarter against itself. Neither case involved a repeated prior answer or a made-up number, and both came down to the same hardcoded default that the model never learned to override.

The tool itself was never the constraint here. Its Time_Period argument already takes a list, so one call naming both quarters would have worked. A separate batch of training examples shows the same tool correctly swapping to a single named quarter in one call whenever the system default pointed somewhere else. The model had the request in front of it and just left the default value in place instead of replacing it.

That's a related unseen fiscal-period request case: the scored example uses a multi-quarter request (a list in Time_Period), not the atomic single-value fiscal-period transition probe the audit emphasizes. With zero atomic fiscal-period transition demonstrations in training, the model kept the hardcoded default on the closest scored cases andnot replaying an old fiscal year or inventing numbers. The exact atomic transition still needs to be run and scored; only that result can confirm whether the model generalizes to it.

The evaluation set already identifies the right cases to examine next

At the unique-edge level, organizational unit, region, metric family, and operation had complete training support for the transitions used in evaluation. Analytical dimension had support in roughly three-quarters of cases. Fiscal period had 0% support because the evaluation set contained close to a dozen unique fiscal-period transition edges across a small number of events that did not appear in the authored training transitions.

Those fiscal-period cases are useful generalization probes that have led to one actionable eval result so far. The scored example above shows a closely related unseen fiscal-period request failing outright, and failing the same way every time: falling back to a hardcoded default instead of erroring more gracefully. That result does not establish performance on the dozen atomic transition edges; the next step is to run the exact transition itself and score the seen and unseen labels the same way, instead of guessing from coverage numbers alone.

Design the eval set around each field's coverage level

The training-coverage figures from earlier in this series point to a practical way to plan the next evaluation round before any new data is authored. Each field already falls into one of three coverage tiers, and each tier calls for a different kind of eval case.

Framed this way, the fiscal period goes from being a gap to the most demanding generalization test already sitting in the corpus, and the next eval round can treat it that way on purpose instead of by accident.

What a stronger transition-generalization eval could add

The current eval set already contains useful unseen transitions, but a stronger generalization slice could make the comparison much more deliberate. Rather than relying on whichever unseen transitions happen to appear, the next eval could include matched cases where some state changes were demonstrated in training and comparable valid changes were intentionally held out.

A small targeted set could include:

• several fiscal-period switches in both directions
• unseen analytical-dimension transitions
• transitions that occur several turns after the original state was established
• cases where two fields change at once while the rest of the state must remain stable
• unseen transitions embedded inside recovery scenarios
• matched seen and unseen transition pairs so performance can be compared directly

An instance of fiscal-period switches would be a matched pair of eval turns: one asking for FY25 Q3 to FY25 Q4, a transition seen in training, and one asking for FY25 Q4 to FY26 Q1, the same kind of change but unseen, so the two results can be compared directly.

The matched design turns generalization from an incidental byproduct of the eval set into something directly measurable: whether performance holds steady across a transition regardless of whether that specific change was ever demonstrated in training.

Recommended approach

• Define the valid state values and transitions from the product or task specification.
• Extract the transitions that actually appear in the authored conversations.
• Label evaluation transitions as seen or unseen in training.
• When model results are available, compare performance across seen and unseen valid transitions.

For the fiscal-period case above, that means confirming the transition is valid by specification, marking it unseen since it never appears in training, and running it instead of assuming the outcome. The related scored multi-quarter request suggests, a hardcoded default is a likely failure mode, but it does not establish how the model will handle the atomic transition; only running and scoring that exact transition will say for sure.

Key takeaway: Treat transition coverage as evidence about training exposure, not as a model-performance claim. The most useful next step is to compare seen and unseen valid transitions against actual evaluation outcomes.

‍