PsiBench asks a model this question 492 times, three runs each, against expert-authored patient vignettes. Here is the shape of it, twice. Decide before you scroll.
Piperacillin is a penicillin. With documented anaphylaxis, this order should never pass quietly. Detection like this is the part the field is genuinely good at.
Sensitivity across the 40 published models. Three of them catch every dangerous order in the benchmark. Hold that thought for the next vignette.
Weekly methotrexate is standard therapy for rheumatoid arthritis, with monitoring in place. The dangerous pattern is daily dosing. Systems that fire on the drug name alone train clinicians to dismiss them, and this benchmark counts that against the model.
Specificity runs from 6.1% to 81.8%. Two of the three models with perfect detection alert on more than 92% of everything they see. Perfect detection bought with reflexive alarm is the behaviour behind a 90% override rate.
The 492 scenarios in the live battery are expert-authored and not published. An unpublished battery is what keeps it a test: nothing can train on it, and a score means the system reasoned, not remembered.
See how 40 models score → How grading works →Field figures: Proulx J, Daines B, Barton M, et al., medRxiv, June 2026, doi:10.64898/2026.06.05.26354271, CC-BY 4.0. The two vignettes above are constructed for this concept and are not drawn from the benchmark.