AI faces a tough physics exam: New benchmark reveals the challenge

Large language models have advanced dramatically in recent years, yet when physicists gave them an undergraduate-level test, even the best models were only correct on around one out of every three questions.

Large language models (LLMs) can now draft legal briefs, debug code, and ace high-school exams. However, a new study shows that they remain far from proficient in fundamental physics. An international research team has introduced PhysUniBench, a benchmark of undergraduate-level physics problems designed to test AI reasoning in fundamental science. The results are clear: even state-of-the-art AI systems answered only about one-third of questions correctly.

A new benchmark for scientific reasoning

PhysUniBench contains over 3,300 physics questions across 8 major subfields from classical mechanics to quantum theory. Each question pairs text with a visual diagram or element, reflecting how physics problems often require interpreting figures or graphs. The questions range from multiple-choice conceptual queries to open-ended problems that demand free-form solutions. An iterative model-in-the-loop process filtered out any problems models could already solve, ensuring the benchmark emphasizes genuine failure modes.

Underwhelming performance by top AI models

Even the most sophisticated models fell short of basic proficiency, with overall accuracies ranging from ~17% to ~37%. Certain topics proved especially challenging: none of the models managed to correctly solve open-ended questions in quantum mechanics, apart from GPT-4o, which achieved just 6.2% accuracy in that category. By comparison, a well-prepared undergraduate student would be able to answer a large majority of these questions correctly.

Why physics stumps AI

The challenges highlighted by PhysUniBench reflect fundamental gaps: physics questions require several steps of deduction rather than a single inference; the field relies on algebraic and calculus manipulation at a level of precision AI struggles with; every question includes a diagram requiring visual and contextual understanding; and beyond math, true proficiency requires physical intuition that AI lacks. Current AI models do not have reliable mechanisms to sanity-check their answers, and frequently gave answers that looked plausible but were fundamentally incorrect.

The PhysUniBench results are a reality check against hype, but also a roadmap. Real intelligence in science requires more than crunching data; it requires reasoning through the laws of nature.

This article was created with the assistance of artificial intelligence and thoroughly edited by FirstPrinciples staff and scientific advisors.