Difficulty labels under structural audit
Controlled trends, paired trajectories, and a retrospective comparison of mathematics-item tiers.
A difficulty label and a measured reasoning feature are not interchangeable. These original figures examine how text length, solution structure, and tier assignments relate in one evaluated mathematics bank.
232 generated mathematics items. Tier assignments are intended levels. Correlations, controlled trends, and retrospective relabelling do not establish item difficulty for a human population.
Which item features still track the intended tier after adjustment?
The controlled associations point in different directions. Text length rises with intended tier while several reasoning measures decline.
Hover or focus a point to read the value.
Archived domain/length-controlled audit. n=232 except distractor rationale (n=123). Values reproduce the printed estimates; permutation reference bands are not confidence intervals.
View the plotted values
| Measure | Reported association |
|---|---|
| Solution steps | -0.202 |
| Distinct operations | -0.2 |
| Relation depth | -0.2 |
| Intermediate step | -0.2 |
| Distractor rationale | 0.186 |
| Stem words | 0.604 |
| Stem characters | 0.672 |
Study context
Analysis of 232 generated mathematics items, controlling for domain and length. Grey bands show the source’s permutation reference intervals. These are item-feature relationships, not measured learner difficulty.
Text length and the reported reasoning measures point in different directions.
The reported controlled associations are positive for text-length features and negative for several solution-structure measures. The analysis of this 232-item bank therefore challenges an interpretation in which the tier ordering simply tracks increasing numbers of reasoning steps.
This is a diagnostic of the evaluated bank and the selected feature set. Concept rarity, abstraction, and human familiarity are not exhausted by step counts. The result motivates examining the tier labels, rather than declaring that longer items must be easier or harder for students.
What is increasing when the tier label increases?
The same four tier groups show opposing trends. Switch between solution steps and text length; each keeps its own measurement scale.
Solution-path length
Hover or focus a point to read the value.
Printed tier means from the 232-item audit. The endpoints belong to different item groups; they are not changes within a learner.
View the plotted values
| Intended tier | Solution steps |
|---|---|
| Foundation | 2 |
| Core | 1.73 |
| Advanced | 1.38 |
| Expert | 1.29 |
Text length
Hover or focus a point to read the value.
The same intended-tier groups. Compare with solution steps using separate units, rather than combining incompatible scales.
View the plotted values
| Intended tier | Stem words |
|---|---|
| Foundation | 12.33 |
| Core | 21.14 |
| Advanced | 26.97 |
| Expert | 29.54 |
Study context
The original paired curves compare reported solution steps and stem length across intended tiers. This is a descriptive audit of the evaluated bank, not an official difficulty calibration.
- solution steps
- The decline concerns the recorded step measure. It does not account for every possible source of mathematical difficulty.
- stem length
- The increase concerns the number of words. It is a surface measurement, not a direct observation of cognitive demand.
The text gets longer while the measured solution path gets shorter.
The reported mean solution-step count falls from roughly two to roughly 1.3 while stem length rises across the tier sequence. Reading both together exposes a divergence that a single “difficulty” label would hide.
The result is not evidence that every higher-tier question is cognitively easier. It establishes that, in this bank, the two measured properties do not support the same explanation of the ordering.
What does a retrospective change of labels actually demonstrate?
Each row places two reported correlations on the same axis. The thin connector shows how the association differs under the two groupings; it does not represent change in student performance.
Hover or focus a point to read the value.
Reported correlations for 232 evaluated mathematics items. Retrospective grouping is not independent validation of difficulty for learners.
View the plotted values
| Measure | Original grouping | Retrospective grouping |
|---|---|---|
| Solution steps | -0.23 | 0.52 |
| Relation depth | -0.32 | 0.37 |
| Distinct operations | -0.04 | 0.56 |
| Intermediate step | -0.32 | 0.37 |
| Distractor rationale | 0.01 | 0.46 |
| Stem length | 0.57 | -0.25 |
Study context
The original scatter and reclassification comparison show the relation between a measured depth score and assigned tiers. This retrospective result does not establish improved student outcomes.
- original grouping
- Look for overlap among the coloured tiers rather than assuming that horizontal length separates them.
- regrouping comparison
- The revised bars show coherence with the selected score. They do not show a change in student performance.
A different grouping makes the chosen measure more orderly.
The original labels overlap across the scatter. The revised grouping produces a clearer ordering in the measured score. That contrast is informative about the alignment between a classification and its criterion.
Because the revised labels use the depth measure, a cleaner depth ordering is not independent validation. A separate response-based evaluation would be needed to establish whether the relabelled groups are better calibrated for learners.
Study record
Source record: extended construct-difficulty study, pp. 13 and 15. June 2026.
Interactive charts use verified aggregate values from the reports and archived numerical outputs. Chart notes distinguish these sources; resampling estimates can differ between runs. Where the underlying numerical series is unavailable, the text states the result without reconstructing unverified data. Study context and interpretation notes accompany each analysis. Detailed production procedures and the full internal documents are not included.