theaicatalog
Hugging Face ยท Research

StateBench tests how language models track changes in long documents

Ai2 published StateBench, a benchmark for how well language models keep track of entities whose information changes across long documents. It has four difficulty levels, from values that never change to chains of variable swaps. The authors say four frontier models performed well on the easiest two levels, but accuracy fell sharply on the harder ones, dropping to 13% or lower on the longest general finite problems. In tests with 275-million-parameter models trained from scratch, Gated DeltaNet and hybrid models generalized almost perfectly to sequences of tens of thousands of assignments, while transformers degraded on longer sequences and never learned general finite problems.