ItInsightsWorld Explainer: Private Questions Now Carry 60% of Intelligence Index v5
Benchmark construction, rather than any single ranking, is what makes this release notable: for the first time most of the Artificial Analysis Intelligence Index is scored against material the public cannot see. Today's unveiling of the Artificial Analysis Intelligence Index v5 is the largest upgrade to the Artificial Analysis Intelligence Index since v4 arrived in January. It brings 2 new benchmarks and 3 upgrades, and lifts the share of the Index evaluated on private test sets or held-out answers to 60%.
The same announcement ships a companion series, the Artificial Analysis Progress Index v1, which plots capability on one continuous scale across time.
Inside the v5 evaluation stack
Two names are new arrivals. FrontierCode 2.0 is Cognition's freshly released private benchmark, built around real-world coding work where the grading question is simply whether a repository maintainer would merge the agent's change. The other addition, Terminal-Bench-Science, hands models 70 agentic scientific research tasks run in a terminal environment.
Three established evaluations move up a version. Humanity's Last Exam steps aside for HLE-Diamond, the refined 1,000-question build from the Center for AI Safety (CAIS) and Scale AI, split evenly between reasoning and knowledge questions. AA-Omniscience v1.1 is the product of housekeeping: saturated and public set questions stripped out, fresh questions added, ambiguous or erroneous wording corrected. CritPt v2 reaches the Index through a private, 40-question hard subset reworked by the CritPt authors and executed with a code-execution tool.
Three members were dropped along the way — GDPval-AA v2.1, SciCode and AA-LCR v1.1 — while AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0 remain in the Index.
Category weights: four blocks at 25%
| Category (total weight) | Constituent evaluation | Share inside the category |
|---|---|---|
| Knowledge Work (25%) | AA-Briefcase | 15% |
| Knowledge Work (25%) | AutomationBench-AA | 5% |
| Knowledge Work (25%) | GDP.pdf | 5% |
| Coding (25%) | FrontierCode 2.0 | 15% |
| Coding (25%) | Terminal-Bench 4.0 | 10% |
| Science (25%) | Terminal-Bench-Science | 15% |
| Science (25%) | CritPt v2 hard subset | 10% |
| General (25%) | AA-Omniscience v1.1 | 15% overall: 10% accuracy plus 5% non-hallucination |
| General (25%) | HLE-Diamond | 10% |
Who sits on top of the composite
At the summit stands Claude Opus 5.5 (max) with 58, just ahead of GPT-6 Astra (max) on 57 and GPT-6.1 Sol (max) on 55. Anthropic and OpenAI occupy the first five places; the next entrant, Gemini 4 Argon (high), posts 50.
Everyone else is chasing a wider gap. Kimi K3.1 Pro (weights not yet released) leads the open weights field at 46, and among checkpoints that are actually downloadable, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) reach 32 — a 26-point deficit to the leader that also leaves them below Claude Haiku 5.5 (max) on 36.
Spending per task is far from uniform: across the leading field the spread exceeds 13x. GPT-6.1 Terra (max) is the bargain among the 10 highest-scoring models displayed at $0.96 per Intelligence Index task, while Claude Fable 5.1 (max) is the priciest at $13.04.
Reading the evaluations one by one
FrontierCode 2.0: would the maintainer merge it?
Cognition pushed this private coding benchmark earlier today. Its 132 tasks live inside real open-source repositories, and grading mirrors a practical decision: would a maintainer merge the agent's change. Fail any blocking check and the attempt scores 0. Tasks stay hidden to avoid contamination, and each is run three times in Stirrup, the Artificial Analysis agentic harness.
Claude Opus 5.5 (max) owns the top slot at 46%. Three systems share the next rung on 42% — GPT-6 Astra (max), GPT-6.1 Terra (max) and Claude Sonnet 5.5 (max). DeepSeek V4.1 Flash 1017 (max) is the best open weights result, 28%, sitting 18 points adrift.
Terminal-Bench-Science: experts set the tasks, all tests must pass
The unit of assessment here is a complete scientific research workflow performed in a terminal. Specialists curate and review every task, and credit is awarded only when every test attached to that task passes.
GPT-6 Astra (max) wins with 63%, Claude Opus 5.5 (max) reaches 59% and GPT-6.1 Sol (max) 58%. The tail falls steeply: Kimi K3.1 Pro (weights not yet released) manages 34%, Claude Haiku 5.5 (max) 20%, and DeepSeek V4.1 Flash 1017 (max), the best openly released model, just 13%. Most open weights systems benchmarked in this round land under 10%.
HLE-Diamond: the knowledge half bites harder
Substituted for Humanity's Last Exam inside the Index, HLE-Diamond is the refined 1,000-question version assembled by CAIS (the Center for AI Safety) together with Scale AI. Claude Opus 5.5 (max) leads it at 68%, GPT-6 Astra (max) follows on 64%, and Claude Fable 5.1 (max) lands at 60%.
Broken down by item type, the pattern flips expectations: reasoning is the easier half for frontier systems. Claude Opus 5.5 (max) and GPT-6 Astra (max) both hit 78% on reasoning, against 58% and 50% respectively on knowledge.
AA-Omniscience v1.1: a scrubbed knowledge and hallucination test
Artificial Analysis cleaned up its knowledge and hallucination evaluation: saturated and public set questions out, new questions in, ambiguous or erroneous questions revised. Consequences are mild — results stay mostly aligned with v1, with models averaging 2 points lower on the AA-Omniscience v1.1 Index and a median of 0.6. A recalibrated public subset, AA-Omniscience-Public-v1.1, also appeared on Hugging Face, tuned so accuracy, hallucination rate and overall performance match between public and private halves.
CritPt v2: only the hard subset counts
What feeds the Index is a private hard slice of CritPt v2. From the 70 research-level physics questions, the 30 that seven frontier models solved in all five attempts at launch were excluded, leaving 40 questions answered inside an offline Python sandbox in Stirrup, the Artificial Analysis agentic harness.
GPT-6 Astra takes this one at 72% under high effort, GPT-6.1 Sol next at 69% under max effort. Kimi K3.1 Pro (weights not yet released) is the highest-scoring open weights model at 45%. Claude Opus 5.5 (max), first overall on the composite, records 61%.
Nine evaluations, six different winners
Sweeping the whole Index remains beyond every system measured, and across all nine evaluations v5 shows how much room for progress is left. Claude Opus 5.5 claims three wins; five other models divide the remaining six:
- Claude Opus 5.5 — a triple: FrontierCode 2.0 (46%), HLE-Diamond (68%) and AA-Omniscience v1.1 (48)
- GPT-6 Astra — Terminal-Bench-Science (63%) plus the CritPt v2 hard subset (72%)
- Claude Sonnet 5.5 — AA-Briefcase, 1812 Elo
- Kimi K3.1 Pro — Terminal-Bench 4.0, 67%
- Gemini 4 Argon — AutomationBench-AA, 78%
- Muse Wonder Full — GDP.pdf, 33% all-pass
The open weights distance grows
Comparisons with v4.3 are unflattering. The best open weights score, Kimi K3.1 Pro (weights not yet released) at 46, trails the leader by 12 points, against a 3-point gap before. Restricting attention to released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) score highest at 32, and Claude Haiku 5.5 (max) outscores all of them.
Two benchmarks expose the shortfall most bluntly: Kimi K3.1 Pro posts 37% on HLE-Diamond where Claude Opus 5.5 (max) reaches 68%, and 34% on Terminal-Bench-Science where GPT-6 Astra (max) reaches 63%.
Bill, effort dial and token appetite
Money first. One Intelligence Index task costs $10.04 for Claude Opus 5.5 (max), $6.24 for GPT-6 Astra (max) and $1.27 for GPT-6.1 Sol (max). The last number is the interesting one: GPT-6.1 Sol (max) finishes within 4 points of Claude Opus 5.5 (max) for roughly an eighth of the per-task price.
The effort dial also swings scores harder than in v4.3. Claude Opus 5.5 moves from 35 at low effort to 58 at max, at 10x the bill per Intelligence Index task ($0.98 versus $10.04). Claude Sonnet 5.5 covers 29 points between low and max, up from 20, and the range for GPT-6.1 Sol doubles from 10 to 20 points.
Work demanded per task climbs too. Every model in the measured set generates more output tokens per Intelligence Index task than in v4.3, with the median up two thirds, 35k to 59k.
- Agentic suites dominate the token budget: Terminal-Bench-Science, Terminal-Bench 4.0, AA-Briefcase and FrontierCode 2.0 carry 55% of the Index weight yet a median ~90% of output tokens
- Terminal-Bench-Science alone swallows about a third of output tokens per task, the heaviest single component
Progress Index v1, the time-series sibling
Running in tandem with v5, the Artificial Analysis Progress Index v1 lines up language models released since November 2022 on one capability scale drawn from a wide span of benchmarks.
Treat it as history: the question it answers is how far underlying capability has moved, including between models released years apart. Judging the present frontier and picking a model for current use cases stays with the Intelligence Index, which remains the Artificial Analysis primary measure of current model intelligence. In design terms the release leans on learnings from the Epoch Capabilities Index, with credit to Epoch AI and Anthropic.
Who built the yardsticks behind v5
The instruments summarised above are not in-house products. They come from Cognition; from Steven Dillmann and the Terminal-Bench team; from the Center for AI Safety with Scale AI; and from Minhui Zhu, Minyang Tian, Hao Peng and the CritPt team. Taken together, their work explains why v5 rewards finished, verifiable tasks over recited answers — and why hiding the majority of its questions makes the ranking harder to game.