theaicatalog
Hugging Face ยท Research

Microsoft releases ThinkingBox to grade AI agents on database state

Microsoft has released ThinkingBox, a benchmark that grades AI agents on the backend records and side effects they leave behind, not on the text they generate. ThinkingBox-Bench covers 507 stateful business workflows, and each task runs 20 times against various models. Microsoft says Claude Opus 5.5 leads at 67.16%, and Kimi-K3 is the strongest open-weights model, within a point of GPT-6-Astra. Kimi-K3 passes only 68 of 507 tasks (13.41%) in all 20 attempts, while Claude Opus 5.5 and Claude Opus 5 each pass 241 of 507 (47.53%). The harness and dataset are on Hugging Face.