Test equality across groups

What the unit asks

The draft names three families of fairness measure: avoiding protected characteristics or their proxies, parity of outcome or error, and equal calibration across groups. It is not enough to say the model does not use the protected attribute. A proxy can carry the same information, and a small sample can hide a large gap.

What good practice looks like

Pre-register the groups and the measures. Report the counts per group, the confidence intervals and the groups that are too small to test. Where a gap is found, say whether the fix was data, threshold or a secondary route.

Evidence an assessor asks for

What a good answer sounds like

A good answer to P3-08 names The group analysis table, with counts and intervals., dates it, and says what it does not cover. A policy or a promise is not an answer. If P3-08 has no artefact, the row is not met and the absence is the finding.

Where comparisons commonly fail

• The equality claim rests on the model not using a protected field.
• The only group reported is age, which is the attribute being estimated.
• A group with fifty samples is reported without an interval.

Questions to put to the supplier

1. Which groups are missing from the sample, and why?
2. Does the system's error differ by skin tone, gender, age band or device?

How this is used in the kit

P3-08 becomes one row of the comparison register, with an owner, a status and the evidence named above. A status of met asks for the artefact and its date, and the free check uses the same row as an evidence question, so the page and the register share one vocabulary.