Keep the comparison reproducible

What the unit asks

Comparability and repeatability both depend on a record. The draft defines comparability as the extent to which a difference between two statistics can be attributed to a real difference rather than to the method or the passage of time. A comparison without a record cannot tell those apart, and a market that cannot compare claims falls back on brand.

What good practice looks like

Store the raw data, the analysis code, the system version, the test dates and the conditions. Keep a change log. When a supplier changes a model or a threshold, treat the old result as historical and say so.

Evidence an assessor asks for

What a good answer sounds like

A good answer to P3-16 names The reproducibility pack: data, code, method and version., dates it, and says what it does not cover. A policy or a promise is not an answer. If P3-16 has no artefact, the row is not met and the absence is the finding.

Where comparisons commonly fail

• The result is published with no method and no data.
• A model update is made during the comparison period and not recorded.
• The analysis code is lost when the analyst leaves.

Questions to put to the supplier

1. Could someone else repeat this comparison?
2. What changed between the two results, and how much of the difference is that change?

How this is used in the kit

P3-16 becomes one row of the comparison register, with an owner, a status and the evidence named above. A status of met asks for the artefact and its date, and the free check uses the same row as an evidence question, so the page and the register share one vocabulary.