AI fluency practical assessment: reproducibility

What this reference covers

Reproducibility means checking how much the same submission changes when scored again. This English edition explains the method and latest published baseline in the Korean reference reviewed on September 17, 2026. The Korean document also contains historical run tables. These results describe the stated scorer version, not every future version of the service.

Latest published baseline

The full-pipeline run on September 8, 2026 used scorer aif-7.2+aif-v-7.3+aif-s-7.0+r-2586e171+p-5d0dea2f: seven simulated sessions, each repeated three times, with 21 valid runs out of 21 attempts. The sessions were generated by models acting as candidates. They are not a sample of human applicants.

The percentages above are measured agreement within this simulated corpus: compare all pairs of raw scores within each session, then average the session agreement rates. They are not accuracy rates against a human hiring decision.

The overall-score stability target was not met. Two of seven sessions had a maximum-to-minimum overall-score range greater than five points: a measured 28.6%, compared with an internal acceptance target of at most 5%. The largest range was 15 points. The unscored rate was a measured 0% (0 of 21 runs), within the internal target of at most 2%. Passing the individual-dimension targets does not cancel the failure on overall-score variation.

The mean pairwise Jaccard similarity of credited findings was 0.968. Score caps changed between runs in three of seven sessions. Mean latency was 140 seconds per run; the maximum was 199 seconds. These observations concern the dated baseline, not a production service-level guarantee.

What is replayed

Narrative feedback is not replayed in this score-reproducibility measurement. Layer 2 alone cannot establish full-pipeline stability because it freezes findings that can activate score caps. The September 4 baseline used an older scorer; it must not be presented as evidence for the current baseline version. The September 6 run used the same seven-session corpus but only the fixed-ledger scoring layer.

How to read the measurements

Exact agreement compares raw dimension scores before caps. For each session, it is the number of matching pairs divided by all valid run pairs, averaged across sessions. Missing scores are treated as a distinct value; two missing values match. Sessions with fewer than two valid runs cannot contribute a pair.

Raw-score SD is the sample standard deviation within a session, averaged across sessions with enough scored runs. The overall-score range is the maximum minus the minimum among valid repetitions. The unscored rate includes both runs marked unscored and attempts that produced no score. Jaccard similarity is the intersection of credited finding IDs divided by their union, averaged over run pairs and sessions.

Acceptance percentages are internal targets set before evaluation. They are not measured human norms, a candidate pass mark, or a promise of future performance. A baseline must be read with its scorer version, corpus size, repetition count and scoring policy.

Different tasks and simulated groups

Different tasks contain different defects. Statistical equating using item-response models has not been established; compare results only for the same task ID. The published process calls for checking task-level distributions after at least 30 sessions per task. Empty tables mean that measurements have not been published, not that no difference exists.

The September 3 known-groups exercise used eight simulated personas once each. It was preliminary and did not meet all design criteria. It is not evidence about the distribution, hiring performance or predictive validity of human candidates. The simulation-distribution and fairness sections have no published results in this reviewed reference.

Fairness, practice and pending evidence

The planned fairness checks suppress group cells smaller than 30. The four-fifths ratio is a screening heuristic, not a legal conclusion. Protected attributes such as gender and age are not collected for these calculations, so fairness on those dimensions has not been measured here.

Personal practice and institutional assessments use separate task pools. In the reviewed September reference, English and Japanese tasks are available for personal practice, while institutional and employer assessment pools remain Korean-only. Publishing this translation does not expand assessment availability.

There is no published estimate of whether prior paid practice improves sealed-assessment results. Planned internal, public and employer pilots are not completed outcomes. The concurrent-load table is also empty: 20 simultaneous sessions is a design target, not verified capacity.

Using the evidence

Read the dimension evidence and limitations before relying on an overall score. Do not use these simulated measurements to rank applicants, predict hiring success or claim that all acceptance criteria have been met. The assessment supports human review and follow-up questions; the institution remains responsible for its decisions.

Read the measurement scope. Open the Korean reference and complete historical tables.