Artificial Analysis overhauls the benchmark everyone uses to judge AI models, and the reshuffle shows a clearer frontier, a wider open-weights gap, and Google suddenly back on top with Gemini 4 Carbon. Plus: blowout cloud quarters, a CapEx plateau the bears didn't predict, and a much bigger DOJ probe of Nvidia.
Benchmarks are a cat-and-mouse game with reality, and the new scoreboard finally matches the vibes.
Benchmarks are inherently limited but important. Without them we only have vibes. Artificial Analysis's Intelligence Index V5 is the latest move in a constant chase to match how people actually use AI. It shifts to mostly private datasets to blunt benchmark-maxing, drops long-context tests that compaction made routine, and goes all-in on agentic coding, agentic science, and granular white-collar work. The result reshuffles the leaderboard. A handful of US labs pull clear of everyone else, the open-weights gap jumps from 3 points to 12, and Google's Gemini 4 Carbon takes first place. Most people say the new rankings finally match their own experience. The open questions are weighting, effort settings, and whether we should stare at one headline number at all.
Without benchmarks, there's nothing but vibes to tell models apart. Yet almost never should one score, or even the aggregate, be taken as gospel. Done well, they're a guide to where different models fit into your stack.
When LLMs were glorified search and writing assistants, the Chatbot Arena ranking was the pinnacle test. Coding and research brought objective benchmarks, then harder versions as each one saturated. The index has repeatedly lagged reality. The early V4.2 update came because GPT-6 Astra's computer-use gains weren't showing up in its score.
For the first time, most benchmarks in the Intelligence Index are built on private datasets. Benchmark-maxing means getting test questions and answers into training data, so a score reflects memory rather than capability. Private data makes it harder to train to the test.
V5 adds Cognition's Frontier Code 2.0, which scores on whether a maintainer would merge the code, plus Terminal Bench Science. It swaps GDPVal for granular white-collar tests and drops the long-context reasoning test. In January long context was critical, but compaction and memory gains mean it's now taken for granted.
The index now weights four categories equally: knowledge work, coding, science, and general. Coding benchmarks measure only agentic performance, and science tests real agentic research rather than stored knowledge. Science's quarter share seems to bet on the science and math race Anthropic and OpenAI have been trading headlines over.
Before Carbon, Opus 5.5 led at 58 and GPT-6 Astra jumped from sixth to second at 57. OpenAI's Sol rose from ninth to third and GPT-6.1 Terra from tenth to fifth. Sonnet 5.5 slipped to fourth, Meta Wonder fell from third to sixth, and Kimi K3.1 Pro dropped from fourth to ninth.
Kimi K3.1 Pro is the top open-weights model at 46. DeepSeek V4.1 Flash and GLM 5.3 sit at 32, below Claude Haiku 5.5's 36. Anthropic, OpenAI, Google, and Meta all have models at 50 or above, followed by a 10-point cliff. Adding more agentic tasks drew a clear line between the frontier and everyone else.
GPT-6.1 Terra is the cheapest top-10 model at $0.96 per task, and Fable 5.1 the priciest at $13.04. Among comparably sized models, Opus and Sonnet run around $10 and GPT-6 Astra $6.24. Gemini 4 Carbon, Grok 4.7, Wonder, and Muse land in the high-$7 range.
The dominant reaction was that V5 tracks subjective experience far better. OpenAI's Dean Ball said it's the first AA ranking to roughly approximate his own. Others noted it confirms a deeper Anthropic/OpenAI lead and that no "Fable-class" open model has appeared yet.
Brookings' Kyle Chan noted private benchmarks only rose from 45% to 60% of the index. Kimi leads all models on Terminal Bench 4.0 at 67% and beats Claude on private AutomationBench, but scores 34% on Terminal Bench Science versus Astra's 63%. His read: this isn't necessarily benchmark gaming, but a real gap in advanced scientific reasoning.
OpenAI and Anthropic are clearly emphasizing scientific discovery. For the average user, though, does science matter as much as agentic coding? That argues either for a different weighting or for skipping the headline number and focusing on the benchmarks for your own use cases.
AA does factor effort into its cost analysis. On a practical level, though, NLW thinks people would be better served by sorting the index by effort setting as well as by model.
AA's new long-term index tracks how each lab's leading model improves over time. People were quick to mine it. One analysis said OpenAI's models lean hardest on reasoning, and another estimated the US–China gap at 3.3 months and closing.
It's one of the shortest gaps yet between versions of ultra-large models. With a new base model in hand, Google can iterate quickly through reinforcement learning instead of waiting on a lengthy pre-training run.
— Sundar Pichai, Google CEO, on the earnings call. On Wednesday's earnings call, Pichai said Google is at the point where AI is helping increase its training and launch velocity. He framed Carbon's quick arrival as evidence of "incredible model momentum."
In coding, Carbon gained 8 points on TerminalBench 4.0 and 6 on Frontier Suite. In cybersecurity, it added 12 on Exploit Gym. In computer use, it picked up 7 on OSWorld and 8 on Agent's Last Exam. It leads the new AA Index V5 at 59, ahead of Opus 5.5, and ties Sonnet 5.5 for second on the AA Coding Index.
When Argon shipped, some felt Google had missed its window, leaving it in a second tier alongside Meta. If real-world use matches the benchmarks, Carbon puts Google genuinely back at the frontier. It does so at a middle-of-the-road $7.76 per task.
Google says it is following the administration's pre-release model testing guidelines and taking a "deliberate approach." A subset of cyber defenders and trusted partners get access first. Developers, businesses, and consumers wait while Google iterates on guardrails, so there are few first-hand testimonials yet.
Arena's render and game-design review of Carbon was slammed for looking less polished than Opus 5.5. NLW isn't convinced 3D design is the right yardstick for LLMs, however much social media fixates on it. Arena's Peter Gastev still said Carbon feels "a lot better than Argon."
Gemini post-training researcher Xiao Zhang contrasted Carbon with the big-bang celebration around Gemini 3 Pro. The team has been dogfooding for weeks: overnight loops running dozens of experiments, root-causing bugs, and swarms inspecting RL data. DeepMind researchers also hinted that an even stronger unreleased Gemini is waiting.
Google has been out of the early-adopter AI conversation for about 10 months. Even so, builders are already reopening Antigravity and calling it "comeback season." NLW's stance is to hold judgment until people get their hands on the model, though it looks promising.
Microsoft broke out Azure for the first time: $33B, up 48%, driving 19% overall growth to $92.3B. OpenAI contributed $10B of Azure revenue. The stock hit a new all-time high after hours, up as much as 4.8%.
Google Cloud doubled to $30.7B, nearly closing the gap with Microsoft. Search growth slowed to 12% from 17%, though, and YouTube to 9.2%. Pichai acknowledged that AI Mode's lower monetization is weighing on results. Some analysts see Google forced to cannibalize Search, and the stock fell as much as 10% after hours.
Google, Microsoft, Oracle, and Meta are all spending below plan this quarter, even as Google guides CapEx up 50% to $300B next year. Bears expected faltering demand to prick the bubble. Instead, hyperscalers appear to want to keep accelerating and are running into components, power, labor, and community support.
Reuters reports DOJ questions about whether owning Hugging Face, a $13B deal, would let Nvidia control a key AI model distribution point. The multi-year probe predates the Groq acqui-hire. It also covers the traditional Run:ai acquisition and another data-center deal, plus venture investments, sales practices, and customer pressure. No charges have been filed.
