Signal Detection for Attribute MSA: Judgement Effectiveness Online Tool
What is Signal Detection?
Attribute measurement systems (pass/fail, defect present/absent) cannot be quantified with GRR, so signal detection is used instead: each appraiser's judgement is compared with a reference value and the miss rate (nonconforming accepted), false rate (conforming rejected), effectiveness and Kappa are tallied to decide whether the system reliably identifies nonconforming units. Signal detection checks whether judgement results align with reference values and is the standard alternative when GRR cannot be applied to attribute data.
How to Design the Study
Select a sample set in which a known proportion of units (e.g., 50%) is nonconforming, and include plenty of borderline parts near the specification limit to challenge the decision boundary. Have each appraiser judge independently and record the results; running multiple rounds also assesses within-appraiser consistency. Reference values must be established by an authoritative method (precision instrument or expert judgement), and part numbering and order should be randomized.
Reading the Results
Effectiveness equals correct decisions divided by total decisions; AIAG accepts an overall effectiveness of at least 90% and conditionally accepts 80% to 90%. The miss rate (false accept) carries more risk than the false rate (false reject) and should be controlled first, and Kappa should be at least 0.75. The tool also outputs a misclassification matrix by part category and per-appraiser comparisons so judging criteria and training can be targeted, and the records of parts with concentrated misses can be traced to clarify ambiguous criteria before revising the work instruction.
Things to Watch
Sample composition drives the result: the higher the share of borderline parts, the harder the judgement and the worse the metrics - a normal outcome that should be stated in the report. A high miss rate lets nonconforming units flow downstream, so retrain appraisers, refine the judging criteria and re-measure. If possible, retest signal detection periodically (e.g., quarterly) to monitor whether judging capability drifts with staff turnover and standard changes.