Understand how students learn.
Change how we teach.

Explore the relationships between a question, an observed response, and what we can responsibly conclude.

Challenged by AI.
Refined by experts.

Each investigation begins with a question. The figure records a test; the interpretation states what that test does—and does not—establish.

How much of the cited evidence is individually necessary?

The graph compares the recorded depth groups with their reported mean evidence dependence. Follow the purple series against the grey diagonal; focus or hover on a point to read its value.

AnnotatedIndividually necessary · mean

Hover or focus a point to read the value.

Reported group means from 16 model-evaluated items. Individual scatter and uncertainty are not reconstructed from the image.

View the plotted values

Annotated depth	Annotated	Individually necessary · mean
4	4	2
5	5	2.3
6	6	2.4

Study context

Sixteen model-evaluated reading items. The scatter preserves the individual observations and reported mean trend. The source’s ceiling language describes this tested setting, not a universal cognitive limit.

The average dependency rises much less than the annotation.

The reported group means are 2.0, 2.3, and 2.4 for annotations of four, five, and six. In the supplementary original figure, individual points spread widely, including zero-valued cases. The relationship is therefore neither one-to-one nor adequately described by the annotation alone.

This changes the interpretation of the preceding calibration curve: an item can carry a larger annotation without a proportionate increase in this operational measure of necessity. Redundancy and the effects of deletion remain possible explanations; the figure does not establish a universal ceiling on cognition.

Continue through the related depth experiments ↗

What separates the observed TOEFL item groups in a low-dimensional view?

Each point is one assessment item. Switch between the spatial view and two planar projections to examine how separation and overlap depend on the view.

Listening · 236Reading · 150

386 items. Coordinates recomputed from the six source files using the recovered analysis. The item count and rounded explained variance match the archived run: 43%, 22%, 13%. These axes describe item features, not learner ability.

Study context

A surface-feature projection from six full-test evaluations: 386 multiple-choice items. Points describe assessment artifacts, not learner states or learning trajectories.

The sections occupy different regions, with overlap still present.

The cloud shows structured separation rather than a single undifferentiated mass. That pattern is consistent with differences in task form and recorded item properties across the sampled sections.

The axes are not learner ability or growth. This corpus contains 386 multiple-choice items from six full-test evaluations. The projection is a view of those artifacts and should not be read as a psychological map of students.

How many dimensions are present—and by which criterion?

The observed spectrum falls below its reference after the third component. Read that boundary first, then compare it with the reconstruction and stability studies below.

ObservedParallel reference · 95th percentile

Hover or focus a point to read the value.

190 generated items. First eight stored components of the text-feature analysis. Three exceed the parallel-analysis reference; this criterion is distinct from variance coverage and stability.

View the plotted values

Component	Observed	Parallel reference · 95th percentile
1	3.496	1.524
2	1.538	1.371
3	1.394	1.263
4	1.111	1.174
5	0.844	1.104
6	0.831	1.039
7	0.715	0.974
8	0.498	0.919

Study context

Analysis of 190 generated reading and writing items. Parallel analysis and cumulative variance describe the feature representation; component retention is not the same as bootstrap stability.

spectrum against references: Compare the observed eigenvalues with the parallel-analysis and broken-stick curves. The reference matters as much as the shape of the scree plot.
cumulative variance: Read the added variance as components accumulate. Reaching a variance threshold does not establish that each included direction is reproducible.

A dominant component does not settle the dimensionality question.

The report identifies a dominant first eigenvalue, retains three components under parallel analysis, and reaches 90% cumulative variance at eight components. Retention above a reference and coverage of total variance are not equivalent criteria.

The next step is to ask whether these directions recur when the sample changes and whether they reconstruct held-out observations. Without those checks, a convenient projection can be mistaken for stable structure.

Explore the spatial view, spectrum and stability ↗

What is increasing when the tier label increases?

The same four tier groups show opposing trends. Switch between solution steps and text length; each keeps its own measurement scale.

Solution-path length

Solution steps

Hover or focus a point to read the value.

Printed tier means from the 232-item audit. The endpoints belong to different item groups; they are not changes within a learner.

View the plotted values

Intended tier	Solution steps
Foundation	2
Core	1.73
Advanced	1.38
Expert	1.29

Text length

Stem words

Hover or focus a point to read the value.

The same intended-tier groups. Compare with solution steps using separate units, rather than combining incompatible scales.

View the plotted values

Intended tier	Stem words
Foundation	12.33
Core	21.14
Advanced	26.97
Expert	29.54

Study context

The original paired curves compare reported solution steps and stem length across intended tiers. This is a descriptive audit of the evaluated bank, not an official difficulty calibration.

solution steps: The decline concerns the recorded step measure. It does not account for every possible source of mathematical difficulty.
stem length: The increase concerns the number of words. It is a surface measurement, not a direct observation of cognitive demand.

The text gets longer while the measured solution path gets shorter.

The reported mean solution-step count falls from roughly two to roughly 1.3 while stem length rises across the tier sequence. Reading both together exposes a divergence that a single “difficulty” label would hide.

The result is not evidence that every higher-tier question is cognitively easier. It establishes that, in this bank, the two measured properties do not support the same explanation of the ordering.

Read the controlled trends and retrospective comparison ↗

Where does the simulated population concentrate?

Colour records density on a logarithmic scale. The bright ridge contains more simulated observations than the darker perimeter.

Hover a cell to read its value.

Recomputed from the recovered simulation code: 30,000 synthetic learners, 3.6 million responses, seed 7. This is a new smaller replay, not the archived 3-million-learner result. Colour: log-scaled count, 1–409.

Low densityHigh density

A dense projected region does not by itself show that the planted states can be distinguished.

Related study

The archived stress run used 3,000,000 simulated learners and 360,000,000 responses. The graph shown here is a separate 30,000-learner replay of the recovered code. Its coordinates and estimates are not the archived run’s measurements.

Both experiments examine behaviour under a specified simulator. Increasing the population tests computational scale and sampling precision; it does not establish that the model describes real learners.

What becomes visible when the simulated observation is richer than right or wrong?

Three separate questions: can the simulated state be recovered, does repeated observation become consistent, and does recovery depend on the strength of the planted distinction? Select one view at a time.

Recovery under two observation regimes

Simulated recovery

Hover or focus a point to read the value.

30,000 simulated learners; 3.6 million responses. Whiskers are the 2.5th–97.5th bootstrap percentiles of held-out prediction AUC. The process interval collapses to 1 in this run. These are simulated outcomes.

View the plotted values

Measure	Simulated recovery
Accuracy only	0.653 (0.644–0.661)
Process observation	1 (1–1)

Consistency with repeated observation

Repeated-observation agreement

Hover or focus a point to read the value.

Same 30,000-learner simulation. Mean split-half correlation rises with observation count. The archived output retains means only, so no uncertainty band is reconstructed. Consistency does not establish validity.

View the plotted values

Attempts	Repeated-observation agreement
20	0.345
40	0.732
60	0.887
90	0.948
120	0.965

Recovery across simulated depth settings

Process observationAccuracy only

Hover or focus a point to read the value.

Five tested settings in the same simulated model. Depth is a simulator parameter, not a measured learner attribute. The process series remains at 1 under these assumptions; this is not evidence of perfect real-world detection.

View the plotted values

Simulated depth parameter	Process observation	Accuracy only
1	1	0.56
1.6	1	0.6
2.2	1	0.641
2.8	1	0.687
3.4	1	0.719

Study context

Simulation study with a planted latent distinction. The original four-panel figure retains recovery, projected states, repeated-attempt reliability, and exploration distributions. It is not a learner-outcome result.

recovery: Compare the evaluated AUC for the two information regimes. The target state is known because it was simulated.
projected states: The projection displays the planted groups under the process representation. Separation is not independent confirmation of the simulation’s realism.
reliability: Repeated-attempt agreement rises with more observations. Reliability and validity remain different properties.
exploration: The distributions compare the number of simulated items needed to reveal the planted distinction under the two policies.

The simulation separates states that overlap in the accuracy-only view.

The archived recovery estimate is 0.653 for accuracy alone and 1.000 for process observation. Mean split-half agreement rises from 0.345 at 20 attempts to 0.965 at 120. Across five tested depth settings, accuracy-only recovery varies while the process result remains at 1. These are separate properties of the same simulated model.

Recovery is conditional on the simulation that placed the distinction in the data. This demonstrates behaviour under those assumptions, not that the same latent state has been identified in a real learner population.

Continue through channel tests, model failure, and scale ↗

Perspectives

10 perspectives

Measurement

The geometry of observed item features

Spectral structure, reconstruction, stability, and category relationships—read across multiple views.

Study Evaluation

Difficulty labels under structural audit

Controlled trends, paired trajectories, and a retrospective comparison of mathematics-item tiers.

Study Measurement

Observation, recovery, and the limits of simulation

A visual record of channel comparisons, recovery experiments, reliability, failed models, and scale tests.

Study Evaluation

Annotated complexity and evidential necessity

Testing whether the evidence cited by an item is necessary to sustain its answer.

Study Evaluation

Evidence coverage is not a single score

A comparison of support for the key and documented refutations of competing answers.

Study Question architecture

Beyond a defensible answer.

What makes an interpretation admissible—and why a plausible answer is not enough.

Perspective Evaluation

What survives a change of view.

Distinguishing a change in expression from a change in what a task measures.

Perspective Evaluation

Quality resists a single ordering.

Why different dimensions of assessment quality cannot be collapsed without assumptions.

Perspective Measurement

An observation is not an explanation.

The limits of inferring a learner’s understanding from an observed response.

Perspective Human judgment

The final distinction is a human one.

Connecting a technical judgment to the person responsible for its use.

Perspective

Read with the limits in view.

These perspectives describe assessment design problems and interpretive commitments. They are not reports of calibrated difficulty, demonstrated learning gains, or deployment of every formalism discussed.

From research to systems

Understand how students learn.Change how we teach.

Challenged by AI.Refined by experts.

The average dependency rises much less than the annotation.

The sections occupy different regions, with overlap still present.

A dominant component does not settle the dimensionality question.

Solution-path length

Text length

The text gets longer while the measured solution path gets shorter.

Recovery under two observation regimes

Consistency with repeated observation

Recovery across simulated depth settings

The simulation separates states that overlap in the accuracy-only view.

Perspectives

The geometry of observed item features

Difficulty labels under structural audit

Observation, recovery, and the limits of simulation

Annotated complexity and evidential necessity

Evidence coverage is not a single score

Beyond a defensible answer.

What survives a change of view.

Quality resists a single ordering.

An observation is not an explanation.

The final distinction is a human one.

Read with the limits in view.

Understand how students learn.
Change how we teach.

Challenged by AI.
Refined by experts.