New AI models and the benchmark paradox

Z.ai's new open-source model and Harmonic's math-native Aristotle highlight contrasting strategies for AI reasoning, while a new wave of increasingly specialized benchmarks invites us to rethink how we measure progress.

Last week offered a snapshot of AI's evolving ambitions and contradictions. China's Z.ai unveiled its flagship open-source language model, GLM-4.5, alongside a cost-efficient variant. On the other side of the Pacific, Silicon Valley startup Harmonic debuted Aristotle, an AI model designed to deliver formally verified, hallucination-free mathematical reasoning.

GLM-4.5: Open-source generalism with agentic intent

GLM-4.5 is built on a vertical mixture-of-experts (MoE) architecture, selectively activating only 32 billion of its 355 billion parameters during inference. It supports a dual operating mode: a lightweight "non-thinking" mode for fast, simple tasks and a heavyweight "thinking" mode for multi-step reasoning and agentic planning. At less than $0.30 per million tokens output, it sets a new price-performance bar among open models, outcompeting recent releases from DeepSeek and Grok-mini.

Harmonic's Aristotle: Formal correctness in a generative age

Harmonic's Aristotle aspires to formal mathematical rigor, claiming not only to solve problems but to verify its reasoning using formal logic systems like Lean 4. Aristotle recently achieved Gold Medal-level performance at the 2025 International Mathematical Olympiad (IMO). Harmonic isn't claiming to replace broad LLMs — it's betting that in high-stakes technical fields, correctness, not coverage, will define the next frontier.

The benchmark paradox: Evaluating the evaluators

Every week seems to bring a new benchmark, evaluating some slice of a model's capability. The granularity offers precision, but also proliferation. Already, some research teams are calling for meta-benchmarking efforts to evaluate the evaluators, rank the relevance of tasks, and model interdependencies among test domains. Large quality differences exist across benchmarks, with many failing to report statistical significance or ensure replicability of their results.

The frontier is splintering into many paths: open and closed, fluent and rigorous, task-driven and theory-aware. If we want AI to do more than perform, our evaluations must evolve from benchmarks into blueprints — guiding not just what we test, but what we value.

This article was created with the assistance of artificial intelligence and thoroughly edited by FirstPrinciples staff and scientific advisors.