Benchmarks & evals papers
What AI researchers are reading: the papers the Hugging Face community picks each day, and new arXiv submissions in the directions we follow. Titles and abstracts are arXiv’s; every paper links to arXiv.
Pterodactyl: Pushing the Limits of Efficient Automated Translation Evaluation
Evelyn Y. Garland, Carola F. Berger
In this paper, we present 1020 Group's submission to the WMT26 shared task on automated quality evaluation. Our system, Pterodactyl, was engineered specifically for subtasks 2 and 3, that is, segment-level quality score prediction and the detection of error-free segments. Pterodactyl is a highly customizable,…
A Theory of Cross-Validation Variance
Nathaniel S. O'Connell, Jaime L. Speiser
Cross-validation estimates predictive performance by repeatedly splitting the data into training and validation sets, fitting the model on the first and evaluating it on the second. Every observation influences the estimate in two roles, through its own evaluation when it is held out and through the models it helps…
AI Isn't Getting Dumber, We're Getting Pickier: Rising Standards in Human Evaluation of Large Language Models
Zihan Chen, Lei Nico Zheng, Di Zhu
Users often complain that AI assistants such as ChatGPT are getting worse. Some versions do change, but human evaluation leaves a second possibility untested: the people judging may change. We separate the two on Chatbot Arena, a public platform where people compare answers from two anonymous large language models…
LM-SimBench: A Unified Framework, Dataset, and Benchmark for Large Language Model Training Simulation
Yu Dong, Hong Li, Qiang Sun et al.
Evaluating candidate parallelization strategies through distributed training runs makes strategy selection prohibitively expensive at scale. Performance simulation offers a lower cost alternative, yet controlled comparisons among existing simulators remain difficult because they differ in the training stages they…
Ninety-Six Ways to Measure a Scribe: Fragmented Evaluation of Ambient Scribes in Healthcare
Amama Mahmood, Shiye Cao, Laura Fleig et al.
Ambient artificial intelligence (AI) scribes have moved rapidly from experimental pilots to enterprise-wide deployment across health systems, promising reduced documentation burden and improved clinician experience. While research evaluating these tools has expanded quickly, there are no agreed-upon protocols or…
An AI Assistant for Automating Anti-Corruption Expertise of Draft Legislation Based on the TRACE Framework
Serhii Zabolotnii, Alina Tarasova
We present an AI assistant for automating anti-corruption expertise of draft legislation based on the TRACE architecture, designed to screen bills for corruption risks: pinpoint provisions an expert would criticise and name the regulatory defect in each. The pipeline combines a deterministic parsing core,…
DailyWildfireCAN: A Dataset and Benchmark for Next-Day Wildfire Spread Prediction in Canada
Mahacine Ettahri, Hager Radi Abdelwahed, Toluwaleke Olutayo et al.
Wildfires represent a critical ecological and socio-economic challenge in Canada, where vast boreal forests and a warming climate have amplified the frequency and severity of extreme fire seasons. Predicting next-day wildfire spread can support fire management, but requires capturing complex interactions among…
PolyRoute: A Twelve-Language and Code-Mixed Benchmark for Typed-Decision Routing, with Baselines and a Leaderboard
Mohammed Nayeem Khan, Sarun Kumar K S
Typed-decision encoders answer a question about a text in one forward pass, but their behaviour outside English is rarely measured on a fixed yardstick. We release PolyRoute: 322 native-language support messages in 12 languages and 9 writing systems, 156 code-mixed or romanised messages (Hinglish, Spanglish,…
GTAGym: Real AAA Game Tasks Environment for Evaluation and Reinforcement Learning
Jiajun Chen, Shicheng Fang, Jiahao Lu et al.
AAA games offer a rich testbed for evaluating agent perception, reasoning, and control. However, evaluating whether agents can complete tasks in AAA games is challenging and often human-annotated because task outcomes are often difficult to verify automatically, let alone applying it for Reinforcement Learning…
FraudBench: A Legal Evaluation of AI Deception
Kevin Wei, Sumaya Nur Adan, Stephan Llerena et al.
We introduce FraudBench, an evaluation of large language models' (LLMs) propensities to recommend, assist, or directly take actions that would constitute fraud under U.S. law. Our evaluation contains a novel dataset of 159 scenarios based on real-world U.S. court cases and validated by legal experts. We present…
One Table, Many Queries: Benchmarking Generalist Tabular Models Across Generation, Imputation, and Prediction
Yuanrui Wang, Kaifang Li, Xuanyue Li et al.
Generation, imputation, and prediction query different aspects of the same tabular distribution, but are usually evaluated separately. This leaves unclear which capabilities they share, how their training objectives interact, and whether one model can perform all three well. We introduce the One Table Benchmark (),…
MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification
Jiaju Ma, Ashvini Kumar Jindal, Chuhua Wang et al.
Large language models can generate motion graphics from static images and natural language requests, but it remains difficult to evaluate how well models can follow requests that combine spatial, appearance, and temporal requirements. We introduce MotionGraphicsBench, an expert-curated benchmark of 100 instances…
CARTBench: Rubric-Based Evaluation of Shopping Advice for Complex and Ambiguous Requests with Timelines
Yuqin Dai, Zhanwei Zhang, Honghui Yang et al.
Large language model (LLM) shopping assistants now recommend products in multi-turn conversations. However, reliable evaluation faces Data scarcity and Difficulty Standardizing Criteria. To address these obstacles, we introduce CARTBench for CART (Complex and Ambiguous Requests with Timelines). For Data scarcity,…
Human-Centred, Risk-Constrained Evaluation of Financial Document Recognition: Evidence, Critical Errors, and Selective Automation
Subodh Lonkar
Financial document recognition is often assessed using character and word error rates, although these metrics do not express whether a system preserves the financial facts on which people and organisations act. This paper develops a human-centred, risk-constrained framework for evaluating financial document…
CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models
Dengzhe Hou, Lingyu Jiang, Fangzhou Lin et al.
Human cognitive tasks let us compare language models with humans under controlled conditions. Yet a model can reproduce the direction of a human effect while making different errors. CogBattery evaluates 54 open-weight language models on text adaptations of 13 cognitive paradigms, comparing each with human data…