Ai2 releases XYBench to test how AI handles misguided questions
Ai2 has released XYBench, a benchmark of 8,115 real-world questions that tests whether AI assistants spot a misconception behind a request and redirect the user. The questions cover programming, cooking, and home repair, and XYBench won the Best Resource Paper Award at EMNLP 2026. Ai2 says the five models it tested, including Claude Sonnet 4.6 and GPT-5, often offered a solution but kept their answers focused on the user's original mistaken approach.