HDATF
English
Start a conversation
Menu

HDATF RESEARCH / BENCHMARK REPORT

What makes AI ready for real work?

Reading public evidence, testing the complete system, and defining what HDATF must prove next.

HDATF ResearchReport Data snapshot

The finding

Public benchmarks help choose a model and design a test. Readiness for work must be established with the actual files, tools and completion criteria of the system being deployed.

This report interprets archived public results through three practical concerns: usable deliverables, traceable evidence and reliable execution. HDATF has not published its own measured scores for these evaluations. The final chapter therefore sets out an evaluation plan.

73 evaluations and metrics684 archived scores

01 / Reading the scores

Begin with the work you need done.

A benchmark is a repeatable test with a defined task and scoring rule. A reasoning test, a document comparison and a tool-use task each observe a different part of an AI system. Their scores answer different questions.

For a document workflow, success means the requested file can be opened, its evidence can be checked and its content meets the brief. For data analysis, the output must also be reproducible from the supplied data. A general model ranking does not resolve either decision on its own.

We use public results to narrow candidates and identify failure modes worth testing. To decide whether a method belongs in LabChin, we then need a matched task using our own sources, permissions, tools and output checks. This report follows that order.[1]

02 / Usable deliverables

Judge the deliverable against the actual brief.

GDPval makes professional deliverables the object of comparison. Its original Gold study used 220 tasks across 44 occupations, with experts comparing outputs without knowing who produced them. The chart reproduces three archived results from that study.

A win-or-tie rate describes those comparisons. It is not the proportion of jobs an organization can hand over to AI. The task set, reviewer judgment and model configurations define how far the result can be interpreted.[2]

Figure 01

GDPval Gold

Expert comparison: wins + ties (%)

%Higher is better
  1. Claude Opus 4.147.6%
  2. GPT-5 (high)38.8%
  3. o3 (high)34.1%
GDPval Gold / %
Model and configurationScore
Claude Opus 4.1Anthropic47.6%
GPT-5 (high)OpenAI38.8%
o3 (high)OpenAI34.1%

220 gold tasks across 44 occupations. Blind, pairwise expert grading in the original study. These are historical results, not current model rankings.

Source: OpenAI2025-09Original record

For HDATF, the practical question is how much work remains after the file is produced. A review should retain missing requirements, unsupported numbers and the revisions a person had to make. A polished layout is one part of quality; it cannot compensate for incorrect evidence or an unusable file.

03 / Evidence and research

Finding an answer and supporting a report are separate checks.

BrowseComp tests whether a system can locate difficult facts. Parallel's published comparison used a random sample of 100 questions from the 1,266-question set. It reported both accuracy and request cost. The results below therefore describe a sampled, provider-run experiment.[3][4]

Figure 02

BrowseComp / Parallel study

Correct answers (%)

%Higher is better
  1. Parallel Ultra8x58%
  2. Parallel Ultra45%
  3. GPT-5 (high)38%
BrowseComp / Parallel study / %
Model and configurationScore
Parallel Ultra8xParallel58%
Parallel UltraParallel45%
GPT-5 (high)OpenAI38%

Random sample of 100 of 1,266 questions. Measured 11–29 August 2025 by Parallel. Ultra8x: $2.40/request; Ultra: $0.30; GPT-5: $0.488. Configurations and budgets differ.

Source: Parallel2025-09-09Original record

Ultra8x reported 58% accuracy at $2.40 per request; Ultra reported 45% at $0.30. Their budgets differ. These figures make the accuracy-cost tradeoff visible, but do not isolate the effect of the underlying model or prove an advantage at equal cost.

A research report needs additional checks. DeepResearch Bench II and ResearchRubrics examine report content against explicit criteria. For LabChin, we plan to retain the passages supporting each major claim, identify contradictory evidence and record which parts of the question remain unanswered.

04 / Execution and memory

The model works inside a system. Evaluate that system.

An agent harness is the software that supplies a model with tools, context and control over a task. LangChain held GPT-5.2-Codex fixed while changing this surrounding software. Its Terminal-Bench 2.0 result rose from 52.8% to 66.5% across 89 tasks.[5][6]

Figure 03

Terminal-Bench 2.0

Task success (%)

%Higher is better
  1. Deep Agents / improved66.5%
  2. Deep Agents / baseline52.8%
Terminal-Bench 2.0 / %
Model and configurationScore
Deep Agents / improvedLangChain66.5%
Deep Agents / baselineLangChain52.8%

89 tasks. GPT-5.2-Codex held fixed; LangChain changed prompts, tools and middleware. Harbor / Daytona execution. The post reports a 13.7 percentage-point improvement.

Source: LangChain2026-02-17Original record

The study changed instructions, tools and middleware together, including checks before completion and guidance about the environment. The 13.7-point difference belongs to that combined experiment. It does not establish the contribution of each individual change or predict HDATF's improvement.

This suggests a testable choice for our own work: hold the task and model fixed, then compare a workflow with and without output verification. Keep the produced file, verification result, retries and elapsed time. The test should explain both successful completion and the failures that required intervention.

Memory needs a separate test. Recalling an old fact is useful only if the system also handles corrections and keeps information within its permitted scope. The archived LoCoMo result covers single-hop recall; it does not verify these work-level requirements. We therefore keep it as supporting evidence in the appendix.

05 / HDATF evaluation plan

The next evidence must be our own work, measured.

The public studies identify useful questions; they do not supply a score for our product. Our next step is a recorded comparison on representative tasks. We plan to assess evidence retrieval, analysis and usable delivery separately, so a strong result in one area cannot hide a failure in another.

For research and data work, preserve the question, source versions and checks on the final output. A report should expose unsupported claims. An analysis should retain the input data, execution record and reproducible outputs. The following criteria define what we intend to inspect, not completed tests.[1][7][8][9]

Planned evaluation. No HDATF score published.

Find the evidence

BrowseComp tests finding hard-to-locate facts on the web. For LabChin, we also want to check whether each cited source supports the claim made from it.

BrowseComp

Build a sound analysis

DeepResearch Bench II and ResearchRubrics provide criteria for research reports. We plan to check coverage, factual support and the reasoning behind the conclusion, not just fluent writing.

DeepResearch Bench IIResearchRubrics

Deliver usable work

GDPval and Agents’ Last Exam examine professional tasks. We plan to check whether the requested work is complete, the files are usable, and a person still has to repair the result.

GDPvalAgents’ Last Exam

Evaluation scopeKeep as evidenceDecision it supports
TaskBrief, input files and acceptance criteriaDid the requested work get done?
ExecutionModel, tools, permissions and configuration versionCan another run use the same conditions?
OutcomeOutput files, source checks and human correctionsIs the output correct and usable?
EfficiencyElapsed time, tools, retries and total costIs the complete workflow affordable?
APPENDIX

Appendix. The complete evidence

Compare published scores by evaluation. Select a model name to open its source. Results reflect the recorded date; model-family comparisons show the best measured configuration for each family.

73 / 73 evaluations and metrics

Work and documents

AA-Briefcase15 results

Base model resultArtificial Analysis

Long-horizon knowledge work

EloHigher is better
  1. Claude Fable 5.11,661.91
  2. Claude Opus 51,647.02
  3. GPT-6 Astra1,561.95
  4. Muse Spark 1.31,558.85
  5. Grok 4.61,545.82
  6. Claude Fable 51,533.52
  7. GLM-5.31,515.45
  8. Kimi K31,496.83
  9. GPT-5.6 Sol1,474.49
  10. GLM 5.3 Flash1,459.13
  11. Qwen3.8 2.4T A95B1,442.81
  12. DeepSeek V4 Flash 07311,433.23
  13. Qwen3.8 27B1,401.81
  14. Qwen3.8 Max1,392.33
  15. Claude Sonnet 51,358.11
AA-Briefcase / Elo
Model and configurationScore
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic1,661.91
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic1,647.02
GPT-6 Astra (max)OpenAI1,561.95
Muse Spark 1.3 (max)Meta1,558.85
Grok 4.6 (xhigh)SpaceXAI1,545.82
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic1,533.52
GLM-5.3 (max)Z AI1,515.45
Kimi K3 (max)Kimi1,496.83
GPT-5.6 Sol (max)OpenAI1,474.49
GLM 5.3 Flash (GLM-5.3-Flash)Z AI1,459.13
Qwen3.8 2.4T A95BAlibaba1,442.81
DeepSeek V4 Flash 0731 (DeepSeek V4 Flash Vision (Reasoning, Max Effort))DeepSeek1,433.23
Qwen3.8 27B (xhigh)Alibaba1,401.81
Qwen3.8 MaxAlibaba1,392.33
Claude Sonnet 5 (Adaptive Reasoning, Max Effort)Anthropic1,358.11
AA-Briefcase / Rubric15 results

Base model resultArtificial Analysis

Required checks passed

%Higher is better
  1. Claude Fable 5.161.52%
  2. Claude Opus 557.98%
  3. Claude Fable 555.96%
  4. Muse Spark 1.354.87%
  5. Grok 4.653.84%
  6. GPT-6 Astra52.32%
  7. GLM-5.351.35%
  8. Kimi K350.97%
  9. Qwen3.8 2.4T A95B50%
  10. Qwen3.8 Max49.6%
  11. Qwen3.8 27B47.8%
  12. GLM 5.3 Flash46.36%
  13. DeepSeek V4 Flash 073146.15%
  14. Muse Spark 1.245.15%
  15. DeepSeek V4 Pro 081343.52%
AA-Briefcase / Rubric / %
Model and configurationScore
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic61.52%
Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)Anthropic57.98%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic55.96%
Muse Spark 1.3 (max)Meta54.87%
Grok 4.6 (xhigh)SpaceXAI53.84%
GPT-6 Astra (xhigh)OpenAI52.32%
GLM-5.3 (max)Z AI51.35%
Kimi K3 (max)Kimi50.97%
Qwen3.8 2.4T A95BAlibaba50%
Qwen3.8 MaxAlibaba49.6%
Qwen3.8 27B (xhigh)Alibaba47.8%
GLM 5.3 Flash (GLM-5.3-Flash)Z AI46.36%
DeepSeek V4 Flash 0731 (DeepSeek V4 Flash Vision (Reasoning, Max Effort))DeepSeek46.15%
Muse Spark 1.2 (xhigh)Meta45.15%
DeepSeek V4 Pro 0813 (Reasoning, Max Effort)DeepSeek43.52%
AA-Briefcase / Analytical quality15 results

Base model resultArtificial Analysis

Quality of analysis

EloHigher is better
  1. Claude Fable 5.11,959.49
  2. Claude Opus 51,909.74
  3. GLM-5.31,788.62
  4. GPT-6 Astra1,753.64
  5. Muse Spark 1.31,711.52
  6. Grok 4.61,682.58
  7. Claude Fable 51,676.55
  8. Kimi K31,668.89
  9. GLM 5.3 Flash1,650.95
  10. Qwen3.8 2.4T A95B1,617.45
  11. DeepSeek V4 Flash 07311,617.37
  12. Qwen3.8 27B1,590.24
  13. GPT-5.6 Sol1,537.92
  14. Qwen3.8 Max1,533.14
  15. Claude Sonnet 51,418.22
AA-Briefcase / Analytical quality / Elo
Model and configurationScore
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic1,959.49
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic1,909.74
GLM-5.3 (max)Z AI1,788.62
GPT-6 Astra (max)OpenAI1,753.64
Muse Spark 1.3 (max)Meta1,711.52
Grok 4.6 (xhigh)SpaceXAI1,682.58
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic1,676.55
Kimi K3 (max)Kimi1,668.89
GLM 5.3 Flash (GLM-5.3-Flash)Z AI1,650.95
Qwen3.8 2.4T A95BAlibaba1,617.45
DeepSeek V4 Flash 0731 (DeepSeek V4 Flash Vision (Reasoning, Max Effort))DeepSeek1,617.37
Qwen3.8 27B (xhigh)Alibaba1,590.24
GPT-5.6 Sol (max)OpenAI1,537.92
Qwen3.8 MaxAlibaba1,533.14
Claude Sonnet 5 (Adaptive Reasoning, Max Effort)Anthropic1,418.22
AA-Briefcase / Presentation15 results

Base model resultArtificial Analysis

Deliverable presentation

EloHigher is better
  1. GPT-5.6 Sol1,633.93
  2. Claude Opus 51,544.01
  3. GPT-6 Astra1,535.29
  4. GPT-5.6 Terra1,519.13
  5. Grok 4.61,514.68
  6. Muse Spark 1.31,506.83
  7. Claude Fable 5.11,472.7
  8. GPT-5.6 Luna1,471.03
  9. Claude Fable 51,456.83
  10. Kimi K31,435.29
  11. Claude Sonnet 51,409.5
  12. GLM 5.3 Flash1,409.12
  13. DeepSeek V4 Flash 07311,355.99
  14. GLM-5.31,354.5
  15. Muse Spark 1.21,334.35
AA-Briefcase / Presentation / Elo
Model and configurationScore
GPT-5.6 Sol (max)OpenAI1,633.93
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic1,544.01
GPT-6 Astra (max)OpenAI1,535.29
GPT-5.6 Terra (max)OpenAI1,519.13
Grok 4.6 (xhigh)SpaceXAI1,514.68
Muse Spark 1.3 (max)Meta1,506.83
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic1,472.7
GPT-5.6 Luna (max)OpenAI1,471.03
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic1,456.83
Kimi K3 (max)Kimi1,435.29
Claude Sonnet 5 (Adaptive Reasoning, Max Effort)Anthropic1,409.5
GLM 5.3 Flash (GLM-5.3-Flash)Z AI1,409.12
DeepSeek V4 Flash 0731 (DeepSeek V4 Flash Vision (Reasoning, Max Effort))DeepSeek1,355.99
GLM-5.3 (max)Z AI1,354.5
Muse Spark 1.2 (xhigh)Meta1,334.35
GDPval-AA v215 results

Base model resultArtificial Analysis

Real-world professional work

EloHigher is better
  1. Claude Fable 5.11,765.9
  2. Claude Opus 51,738.08
  3. Muse Spark 1.31,719.65
  4. GLM-5.31,678.47
  5. GLM 5.3 Flash1,672.94
  6. Grok 4.61,664.82
  7. Qwen3.8-Flash-Next1,649.93
  8. Claude Fable 51,635.39
  9. Qwen3.8 Max1,633.53
  10. Qwen3.8 2.4T A95B1,630.39
  11. GPT-5.6 Sol1,625.68
  12. Kimi K31,585.72
  13. GPT-6 Astra1,582.24
  14. DeepSeek V4 Flash 07311,579.3
  15. Muse Spark 1.21,525.03
GDPval-AA v2 / Elo
Model and configurationScore
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic1,765.9
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic1,738.08
Muse Spark 1.3 (max)Meta1,719.65
GLM-5.3 (max)Z AI1,678.47
GLM 5.3 Flash (GLM-5.3-Flash)Z AI1,672.94
Grok 4.6 (xhigh)SpaceXAI1,664.82
Qwen3.8-Flash-NextAlibaba1,649.93
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic1,635.39
Qwen3.8 MaxAlibaba1,633.53
Qwen3.8 2.4T A95BAlibaba1,630.39
GPT-5.6 Sol (max)OpenAI1,625.68
Kimi K3 (max)Kimi1,585.72
GPT-6 Astra (max)OpenAI1,582.24
DeepSeek V4 Flash 0731 (DeepSeek V4 Flash Vision (Reasoning, Max Effort))DeepSeek1,579.3
Muse Spark 1.2 (xhigh)Meta1,525.03
AA-AnalystAgent / pass^515 results

Base model resultArtificial Analysis

Reliable spreadsheet and document analysis

%Higher is better
  1. Gemini 3.7 Flash60%
  2. Claude Fable 5.157.5%
  3. Claude Opus 553.75%
  4. GPT-6 Astra51.25%
  5. Claude Fable 548.75%
  6. GPT-5.6 Sol47.5%
  7. Claude Sonnet 546.25%
  8. Gemini 3.1 Pro Preview41.25%
  9. Grok 4.641.25%
  10. Kimi K338.75%
  11. Grok 4.535%
  12. Inkling Small27.5%
  13. Inkling23.75%
  14. MiMo-V2.5-Pro20%
  15. DeepSeek V4 Pro 042418.75%
AA-AnalystAgent / pass^5 / %
Model and configurationScore
Gemini 3.7 Flash (high)Google60%
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic57.5%
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic53.75%
GPT-6 Astra (max)OpenAI51.25%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic48.75%
GPT-5.6 Sol (max)OpenAI47.5%
Claude Sonnet 5 (Adaptive Reasoning, Max Effort)Anthropic46.25%
Gemini 3.1 Pro PreviewGoogle41.25%
Grok 4.6 (high)SpaceXAI41.25%
Kimi K3 (max)Kimi38.75%
Grok 4.5 (high)SpaceXAI35%
Inkling SmallThinking Machines27.5%
Inkling (xhigh)Thinking Machines23.75%
MiMo-V2.5-ProXiaomi20%
DeepSeek V4 Pro 0424 (DeepSeek V4 Pro (Reasoning, Max Effort))DeepSeek18.75%
GDP.pdf / All-pass15 results

Base model resultArtificial Analysis

Professional document reasoning

%Higher is better
  1. GPT-6 Astra33.2%
  2. GPT-5.6 Sol28.2%
  3. Claude Fable 5.128%
  4. Muse Spark 1.325.6%
  5. GPT-5.6 Terra25.6%
  6. Claude Fable 524%
  7. GPT-5.6 Luna23.8%
  8. Gemini 3.7 Flash23.6%
  9. Gemini 3.8 Flash22.8%
  10. Qwen3.8 Max21.8%
  11. Claude Opus 521.6%
  12. GPT-5.5 Instant (June 2026)20.2%
  13. Kimi K319.6%
  14. Grok 4.518.8%
  15. Grok 4.618.8%
GDP.pdf / All-pass / %
Model and configurationScore
GPT-6 Astra (max)OpenAI33.2%
GPT-5.6 Sol (max)OpenAI28.2%
Claude Fable 5.1 (Adaptive Reasoning, Low Effort, Default Fallback)Anthropic28%
Muse Spark 1.3 (max)Meta25.6%
GPT-5.6 Terra (max)OpenAI25.6%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic24%
GPT-5.6 Luna (xhigh)OpenAI23.8%
Gemini 3.7 Flash (high)Google23.6%
Gemini 3.8 Flash (medium)Google22.8%
Qwen3.8 MaxAlibaba21.8%
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic21.6%
GPT-5.5 Instant (June 2026)OpenAI20.2%
Kimi K3 (max)Kimi19.6%
Grok 4.5 (high)SpaceXAI18.8%
Grok 4.6 (high)SpaceXAI18.8%
AutomationBench-AA15 results

Base model resultArtificial Analysis

Objectives completed in SaaS workflows

%Higher is better
  1. Gemini 3.7 Flash62.75%
  2. Kimi K352.71%
  3. Grok 4.551.44%
  4. GPT-5.6 Sol51.19%
  5. Gemini 3.6 Flash51.08%
  6. Gemini 3.8 Flash50.72%
  7. Claude Fable 548.58%
  8. GPT-5.6 Terra45.61%
  9. GPT-5.6 Luna42.24%
  10. Claude Sonnet 539.19%
  11. Gemini 3.1 Pro Preview37.54%
  12. Gemini 3.5 Flash-Lite32.66%
  13. GLM-5.227.84%
  14. Kimi K2.7 Code22.53%
  15. Qwen3.7 Plus20.43%
AutomationBench-AA / %
Model and configurationScore
Gemini 3.7 Flash (high)Google62.75%
Kimi K3 (max)Kimi52.71%
Grok 4.5 (high)SpaceXAI51.44%
GPT-5.6 Sol (max)OpenAI51.19%
Gemini 3.6 Flash (high)Google51.08%
Gemini 3.8 Flash (high)Google50.72%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic48.58%
GPT-5.6 Terra (max)OpenAI45.61%
GPT-5.6 Luna (max)OpenAI42.24%
Claude Sonnet 5 (Adaptive Reasoning, Max Effort)Anthropic39.19%
Gemini 3.1 Pro PreviewGoogle37.54%
Gemini 3.5 Flash-LiteGoogle32.66%
GLM-5.2 (max)Z AI27.84%
Kimi K2.7 CodeKimi22.53%
Qwen3.7 PlusAlibaba20.43%
EnterpriseOps-Gym-AA15 results

Base model resultArtificial Analysis

Enterprise operations

%Higher is better
  1. Claude Fable 551.12%
  2. Gemini 3.7 Flash50.4%
  3. DeepSeek V4 Pro 081349.6%
  4. Grok 4.648.34%
  5. Claude Opus 547.48%
  6. Qwen3.8 2.4T A95B47.36%
  7. Muse Spark 1.247.27%
  8. DeepSeek V4 Flash 073147.09%
  9. Kimi K345.33%
  10. Claude Sonnet 544.67%
  11. Qwen3.8 27B44.23%
  12. GPT-5.6 Sol42.91%
  13. GLM-5.242.73%
  14. Inkling Small42.7%
  15. Gemini 3.5 Flash-Lite42.35%
EnterpriseOps-Gym-AA / %
Model and configurationScore
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic51.12%
Gemini 3.7 Flash (medium)Google50.4%
DeepSeek V4 Pro 0813 (Reasoning, Max Effort)DeepSeek49.6%
Grok 4.6 (high)SpaceXAI48.34%
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic47.48%
Qwen3.8 2.4T A95BAlibaba47.36%
Muse Spark 1.2 (xhigh)Meta47.27%
DeepSeek V4 Flash 0731 (DeepSeek V4 Flash Vision (Reasoning, Max Effort))DeepSeek47.09%
Kimi K3 (max)Kimi45.33%
Claude Sonnet 5 (Adaptive Reasoning, Max Effort)Anthropic44.67%
Qwen3.8 27B (xhigh)Alibaba44.23%
GPT-5.6 Sol (max)OpenAI42.91%
GLM-5.2 (max)Z AI42.73%
Inkling SmallThinking Machines42.7%
Gemini 3.5 Flash-LiteGoogle42.35%
APEX-Agents-AA14 results

Base model resultArtificial Analysis

Long-horizon professional tasks

%Higher is better
  1. Kimi K341.3%
  2. GPT-5.6 Terra38.94%
  3. GPT-5.6 Luna35.84%
  4. GLM-5.233.7%
  5. Gemini 3.1 Pro Preview32.01%
  6. Apodex 1.131.19%
  7. DeepSeek V4 Pro 042424.26%
  8. Qwen3.7 Plus22.42%
  9. Qwen3.5 397B A17B15.34%
  10. Step 3.7 Flash14.82%
  11. gpt-oss-120b3.1%
  12. MiMo-V2.5-Pro2.43%
  13. Nemotron 3 Super 120B A12B1.84%
  14. gpt-oss-20b0.74%
APEX-Agents-AA / %
Model and configurationScore
Kimi K3 (max)Kimi41.3%
GPT-5.6 Terra (max)OpenAI38.94%
GPT-5.6 Luna (max)OpenAI35.84%
GLM-5.2 (max)Z AI33.7%
Gemini 3.1 Pro PreviewGoogle32.01%
Apodex 1.1Apodex31.19%
DeepSeek V4 Pro 0424 (DeepSeek V4 Pro (Reasoning, Max Effort))DeepSeek24.26%
Qwen3.7 PlusAlibaba22.42%
Qwen3.5 397B A17B (Reasoning)Alibaba15.34%
Step 3.7 FlashStepFun14.82%
gpt-oss-120b (high)OpenAI3.1%
MiMo-V2.5-ProXiaomi2.43%
Nemotron 3 Super 120B A12B (Reasoning)NVIDIA1.84%
gpt-oss-20b (high)OpenAI0.74%

System studies

GDPval Gold3 results

Expert-comparison studyOpenAI

Can an AI deliver work that a professional would accept? Expert comparison: wins + ties (%)

%Higher is better
  1. Claude Opus 4.147.6%
  2. GPT-5 (high)38.8%
  3. o3 (high)34.1%
GDPval Gold / %
Model and configurationScore
Claude Opus 4.1Anthropic47.6%
GPT-5 (high)OpenAI38.8%
o3 (high)OpenAI34.1%
BrowseComp / Parallel study3 results

Research-agent studyParallel

Can an agent find and connect hard-to-locate evidence? Correct answers (%)

%Higher is better
  1. Parallel Ultra8x58%
  2. Parallel Ultra45%
  3. GPT-5 (high)38%
BrowseComp / Parallel study / %
Model and configurationScore
Parallel Ultra8xParallel58%
Parallel UltraParallel45%
GPT-5 (high)OpenAI38%
Terminal-Bench 2.02 results

Execution-system studyLangChain

How much can the execution system change the outcome? Task success (%)

%Higher is better
  1. Deep Agents / improved66.5%
  2. Deep Agents / baseline52.8%
Terminal-Bench 2.0 / %
Model and configurationScore
Deep Agents / improvedLangChain66.5%
Deep Agents / baselineLangChain52.8%
LoCoMo / single-hop3 results

Memory-system studyMem0 research team

Does useful information survive across conversations? LLM-as-a-Judge score, single-hop (0–100)

scoreHigher is better
  1. Mem067.13
  2. LangMem62.23
  3. Zep61.7
LoCoMo / single-hop / score
Model and configurationScore
Mem0Mem067.13
LangMemLangChain62.23
ZepZep61.7

Agents and tools

τ³-Banking15 results

Base model resultArtificial Analysis

Banking policies and tool execution

%Higher is better
  1. Muse Spark 1.352.37%
  2. Qwen3.8 Max51.34%
  3. Grok 4.650.72%
  4. GLM-5.350.31%
  5. Qwen3.8 2.4T A95B49.07%
  6. Qwen3.8 27B48.04%
  7. GLM 5.3 Flash47.22%
  8. Claude Fable 5.147.22%
  9. Kimi K345.98%
  10. Gemini 3.8 Flash45.77%
  11. Qwen3.8-Flash-Next45.36%
  12. Claude Opus 544.74%
  13. GPT-5.6 Sol44.33%
  14. GPT-6 Astra43.09%
  15. Grok 4.542.06%
τ³-Banking / %
Model and configurationScore
Muse Spark 1.3 (max)Meta52.37%
Qwen3.8 MaxAlibaba51.34%
Grok 4.6 (high)SpaceXAI50.72%
GLM-5.3 (max)Z AI50.31%
Qwen3.8 2.4T A95BAlibaba49.07%
Qwen3.8 27B (xhigh)Alibaba48.04%
GLM 5.3 Flash (GLM-5.3-Flash)Z AI47.22%
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic47.22%
Kimi K3 (max)Kimi45.98%
Gemini 3.8 Flash (medium)Google45.77%
Qwen3.8-Flash-NextAlibaba45.36%
Claude Opus 5 (Adaptive Reasoning, High Effort)Anthropic44.74%
GPT-5.6 Sol (max)OpenAI44.33%
GPT-6 Astra (xhigh)OpenAI43.09%
Grok 4.5 (high)SpaceXAI42.06%
τ²-Bench15 results

Base model resultArtificial Analysis

Conversational tool use

%Higher is better
  1. JT-35B-Flash99.12%
  2. GLM-5.299.12%
  3. Step 3.7 Flash98.54%
  4. Claude Fable 598.54%
  5. DeepSeek V4 Pro 042496.2%
  6. Qwen3.5 397B A17B95.61%
  7. Gemini 3.5 Flash95.61%
  8. Gemini 3.1 Pro Preview95.61%
  9. Qwen3.6 35B A3B95.32%
  10. DeepSeek V4 Flash 042094.44%
  11. MiMo-V2.5-Pro94.15%
  12. Qwen3.6 27B94.15%
  13. Mistral Medium 3.594.15%
  14. Qwen3.5 122B A10B93.57%
  15. MiMo-V2-Flash (Feb 2026)93.27%
τ²-Bench / %
Model and configurationScore
JT-35B-FlashChina Mobile99.12%
GLM-5.2 (max)Z AI99.12%
Step 3.7 FlashStepFun98.54%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic98.54%
DeepSeek V4 Pro 0424 (DeepSeek V4 Pro (Reasoning, Max Effort))DeepSeek96.2%
Qwen3.5 397B A17B (Reasoning)Alibaba95.61%
Gemini 3.5 Flash (medium)Google95.61%
Gemini 3.1 Pro PreviewGoogle95.61%
Qwen3.6 35B A3B (Reasoning)Alibaba95.32%
DeepSeek V4 Flash 0420 (DeepSeek V4 Flash (Non-reasoning))DeepSeek94.44%
MiMo-V2.5-ProXiaomi94.15%
Qwen3.6 27B (Reasoning)Alibaba94.15%
Mistral Medium 3.5Mistral94.15%
Qwen3.5 122B A10B (Reasoning)Alibaba93.57%
MiMo-V2-Flash (Feb 2026)Xiaomi93.27%
ITBench-AA15 results

Base model resultArtificial Analysis

Incident root-cause analysis

%Higher is better
  1. GPT-5.6 Sol56.21%
  2. GPT-5.6 Terra51.04%
  3. Kimi K347.69%
  4. GLM-5.242.66%
  5. GPT-5.6 Luna40.32%
  6. DeepSeek V4 Pro 042438.32%
  7. MiMo-V2.5-Pro38.23%
  8. Gemma 4 31B37.29%
  9. Qwen3.5 397B A17B34.09%
  10. Gemini 3.1 Pro Preview30.33%
  11. Step 3.7 Flash30.27%
  12. Claude 4.5 Haiku27.31%
  13. Gemma 4 26B A4B23.63%
  14. gpt-oss-120b5.65%
  15. Nemotron 3 Super 120B A12B1.13%
ITBench-AA / %
Model and configurationScore
GPT-5.6 Sol (max)OpenAI56.21%
GPT-5.6 Terra (max)OpenAI51.04%
Kimi K3 (max)Kimi47.69%
GLM-5.2 (max)Z AI42.66%
GPT-5.6 Luna (max)OpenAI40.32%
DeepSeek V4 Pro 0424 (DeepSeek V4 Pro (Reasoning, Max Effort))DeepSeek38.32%
MiMo-V2.5-ProXiaomi38.23%
Gemma 4 31B (Reasoning)Google37.29%
Qwen3.5 397B A17B (Reasoning)Alibaba34.09%
Gemini 3.1 Pro PreviewGoogle30.33%
Step 3.7 FlashStepFun30.27%
Claude 4.5 Haiku (Reasoning)Anthropic27.31%
Gemma 4 26B A4B (Reasoning)Google23.63%
gpt-oss-120b (high)OpenAI5.65%
Nemotron 3 Super 120B A12B (Reasoning)NVIDIA1.13%
MCP-Atlas public set / Z.ai report7 results

Vendor-reported model resultZ.ai (model-card report)

Completing tasks with connected tools.

%Higher is better
  1. GPT-5.268%
  2. GLM-567.8%
  3. Gemini 3 Pro66.6%
  4. Claude Opus 4.565.2%
  5. Kimi K2.563.8%
  6. DeepSeek V3.262.2%
  7. GLM-4.752%
MCP-Atlas public set / Z.ai report / %
Model and configurationScore
GPT-5.2 (xhigh)OpenAI68%
GLM-5 release reportZ.ai67.8%
Gemini 3 Pro (GLM-5 release comparison)Google66.6%
Claude Opus 4.5 (GLM-5 release comparison)Anthropic65.2%
Kimi K2.5 (GLM-5 release comparison)Moonshot AI63.8%
DeepSeek V3.2 (GLM-5 release comparison)DeepSeek62.2%
GLM-4.7 (GLM-5 release comparison)Z.ai52%

Code and execution

Terminal-Bench v2.115 results

Base model resultArtificial Analysis

Terminal task completion

%Higher is better
  1. Claude Fable 5.191.39%
  2. GPT-6 Astra89.89%
  3. GPT-5.6 Sol89.51%
  4. Claude Opus 589.14%
  5. Grok 4.688.39%
  6. GPT-5.6 Terra88.01%
  7. Gemini 3.8 Flash87.64%
  8. Qwen3.8-Flash-Next86.14%
  9. Muse Spark 1.385.77%
  10. Gemini 3.7 Flash85.77%
  11. Kimi K385.02%
  12. Claude Fable 584.64%
  13. GLM 5.3 Flash84.27%
  14. GLM-5.383.9%
  15. Qwen3.8 2.4T A95B82.02%
Terminal-Bench v2.1 / %
Model and configurationScore
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic91.39%
GPT-6 Astra (high)OpenAI89.89%
GPT-5.6 Sol (xhigh)OpenAI89.51%
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic89.14%
Grok 4.6 (high)SpaceXAI88.39%
GPT-5.6 Terra (max)OpenAI88.01%
Gemini 3.8 Flash (high)Google87.64%
Qwen3.8-Flash-NextAlibaba86.14%
Muse Spark 1.3 (max)Meta85.77%
Gemini 3.7 Flash (high)Google85.77%
Kimi K3 (max)Kimi85.02%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic84.64%
GLM 5.3 Flash (GLM-5.3-Flash)Z AI84.27%
GLM-5.3 (max)Z AI83.9%
Qwen3.8 2.4T A95BAlibaba82.02%
SciCode15 results

Base model resultArtificial Analysis

Scientific programming

%Higher is better
  1. Claude Fable 5.163.08%
  2. Claude Fable 561%
  3. Gemini 3.7 Flash59.84%
  4. Muse Spark 1.359.72%
  5. Kimi K359.49%
  6. GLM-5.359.03%
  7. Gemini 3.1 Pro Preview58.68%
  8. GPT-5.6 Sol57.75%
  9. Muse Spark 1.257.41%
  10. Gemini 3.8 Flash56.6%
  11. GPT-6 Astra56.48%
  12. Grok 4.656.48%
  13. Claude Opus 556.37%
  14. Grok 4.554.98%
  15. GPT-5.6 Terra54.98%
SciCode / %
Model and configurationScore
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic63.08%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic61%
Gemini 3.7 Flash (medium)Google59.84%
Muse Spark 1.3 (xhigh)Meta59.72%
Kimi K3 (max)Kimi59.49%
GLM-5.3 (max)Z AI59.03%
Gemini 3.1 Pro PreviewGoogle58.68%
GPT-5.6 Sol (high)OpenAI57.75%
Muse Spark 1.2 (xhigh)Meta57.41%
Gemini 3.8 Flash (high)Google56.6%
GPT-6 Astra (max)OpenAI56.48%
Grok 4.6 (high)SpaceXAI56.48%
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic56.37%
Grok 4.5 (high)SpaceXAI54.98%
GPT-5.6 Terra (max)OpenAI54.98%
LiveCodeBench15 results

Base model resultArtificial Analysis

Competitive programming

%Higher is better
  1. gpt-oss-120b87.83%
  2. ERNIE 5.0 Thinking Preview81.16%
  3. o380.85%
  4. Apriel-v1.6-15B-Thinker80.74%
  5. Qwen3 Next 80B A3B (Reasoning)78.41%
  6. INTELLECT-377.67%
  7. gpt-oss-20b77.67%
  8. K-EXAONE76.83%
  9. Doubao Seed Code76.61%
  10. Magistral Medium 1.275.03%
  11. EXAONE 4.0 32B74.71%
  12. NVIDIA Nemotron 3 Nano 30B A3B74.07%
  13. Llama Nemotron Super 49B v1.573.65%
  14. Nova 2.0 Pro Preview73.02%
  15. Falcon-H1R-7B72.38%
LiveCodeBench / %
Model and configurationScore
gpt-oss-120b (high)OpenAI87.83%
ERNIE 5.0 Thinking PreviewBaidu81.16%
o3OpenAI80.85%
Apriel-v1.6-15B-ThinkerServiceNow80.74%
Qwen3 Next 80B A3B (Reasoning)Alibaba78.41%
INTELLECT-3Prime Intellect77.67%
gpt-oss-20b (high)OpenAI77.67%
K-EXAONE (Reasoning)LG AI Research76.83%
Doubao Seed CodeByteDance Seed76.61%
Magistral Medium 1.2Mistral75.03%
EXAONE 4.0 32B (Reasoning)LG AI Research74.71%
NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)NVIDIA74.07%
Llama Nemotron Super 49B v1.5 (Reasoning)NVIDIA73.65%
Nova 2.0 Pro Preview (medium)Amazon73.02%
Falcon-H1R-7BTII UAE72.38%
SciCode / MiniMax report7 results

Vendor-reported model resultMiniMax (internal evaluation)

Scientific programming tasks.

scoreHigher is better
  1. Gemini 3 Pro56
  2. Claude Opus 4.652
  3. GPT-5.252
  4. Claude Opus 4.550
  5. Claude Sonnet 4.545
  6. MiniMax-M2.544.4
  7. MiniMax-M2.141
SciCode / MiniMax report / score
Model and configurationScore
Gemini 3 Pro (MiniMax internal evaluation)Google56
Claude Opus 4.6 (MiniMax internal evaluation)Anthropic52
GPT-5.2 (Thinking; MiniMax internal evaluation)OpenAI52
Claude Opus 4.5 (MiniMax internal evaluation)Anthropic50
Claude Sonnet 4.5 (MiniMax internal evaluation)Anthropic45
MiniMax-M2.5 (MiniMax internal evaluation)MiniMax44.4
MiniMax-M2.1 (MiniMax internal evaluation)MiniMax41
Terminal Bench 2.1 / DeepSeek GA report8 results

Vendor-reported model resultDeepSeek (GA model-card report)

Coding and tool use in terminal tasks.

scoreHigher is better
  1. Kimi K388.3
  2. Fable-5 (w/ fallback)88
  3. DeepSeek-V4-Pro-081387.9
  4. Opus-4.885
  5. DeepSeek-V4-Flash-073182.7
  6. GLM-5.281
  7. DeepSeek-V4-Pro (Preview)72.1
  8. DeepSeek-V4-Flash (Preview)61.8
Terminal Bench 2.1 / DeepSeek GA report / score
Model and configurationScore
Kimi K3 (Code-agent evaluation; harness and reasoning effort not disclosed for this model)Moonshot AI88.3
Fable-5 (w/ fallback) (Code-agent evaluation; harness and reasoning effort not disclosed for this model; with fallback, details unspecified)Anthropic88
DeepSeek-V4-Pro-0813 (DeepSeek Harness minimal; max reasoning effort; temperature=1.0; top_p=0.95)DeepSeek87.9
Opus-4.8 (Code-agent evaluation; harness and reasoning effort not disclosed for this model)Anthropic85
DeepSeek-V4-Flash-0731 (Code-agent evaluation; harness and reasoning effort not disclosed for this model)DeepSeek82.7
GLM-5.2 (Code-agent evaluation; harness and reasoning effort not disclosed for this model)Z.ai81
DeepSeek-V4-Pro (Preview) (Code-agent evaluation; harness and reasoning effort not disclosed for this model)DeepSeek72.1
DeepSeek-V4-Flash (Preview) (Code-agent evaluation; harness and reasoning effort not disclosed for this model)DeepSeek61.8
SWE-bench Pro refined set / Qwen3.8-Max report5 results

Vendor-reported model resultQwen (model-card report)

Code fixes on Qwen's corrected SWE-bench Pro evaluation set.

scoreHigher is better
  1. Fable 580
  2. Opus 4.869.2
  3. Qwen3.8-Max67.7
  4. GPT 5.6 Sol (max)64.6
  5. Qwen3.7-Max60.6
SWE-bench Pro refined set / Qwen3.8-Max report / score
Model and configurationScore
Fable 5 (Qwen-refined set; Claude Code; temperature=1.0; top_p=0.95; 256K context; may involve fallback; reasoning effort not disclosed)Anthropic80
Opus 4.8 (Qwen-refined set; Claude Code; temperature=1.0; top_p=0.95; 256K context; reasoning effort not disclosed)Anthropic69.2
Qwen3.8-Max service model; Qwen-refined set; Claude Code; temperature=1.0; top_p=0.95; 256K context; reasoning effort not disclosedQwen67.7
GPT 5.6 Sol (max) (Qwen-refined set; Claude Code; max (table header); temperature=1.0; top_p=0.95; 256K context)OpenAI64.6
Qwen3.7-Max (Qwen-refined set; Claude Code; temperature=1.0; top_p=0.95; 256K context; reasoning effort not disclosed)Qwen60.6
SWE-bench Verified / Google G31 (2026-02-19)5 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.1 Pro release comparison)

Repository issue resolution by a model and its coding-agent environment.

%Higher is better
  1. Claude Opus 4.680.8%
  2. Gemini 3.1 Pro80.6%
  3. GPT-5.280%
  4. Claude Sonnet 4.679.6%
  5. Gemini 3 Pro76.2%
SWE-bench Verified / Google G31 (2026-02-19) / %
Model and configurationScore
Claude Opus 4.6 (Thinking (Max); Single attempt; provider-specific scaffolding)Anthropic80.8%
Gemini 3.1 Pro (Thinking (High); Single attempt; provider-specific scaffolding)Google80.6%
GPT-5.2 (Thinking (xhigh); Single attempt; provider-specific scaffolding)OpenAI80%
Claude Sonnet 4.6 (Thinking (Max); Single attempt; provider-specific scaffolding)Anthropic79.6%
Gemini 3 Pro (Thinking (High); Single attempt; provider-specific scaffolding)Google76.2%
SWE-bench Pro (Public) / Google G31 (2026-02-19)4 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.1 Pro release comparison)

Public SWE-bench Pro software-engineering tasks.

%Higher is better
  1. GPT-5.3-Codex56.8%
  2. GPT-5.255.6%
  3. Gemini 3.1 Pro54.2%
  4. Gemini 3 Pro43.3%
SWE-bench Pro (Public) / Google G31 (2026-02-19) / %
Model and configurationScore
GPT-5.3-Codex (Thinking (xhigh); Public split; single attempt; provider-specific scaffolding)OpenAI56.8%
GPT-5.2 (Thinking (xhigh); Public split; single attempt; provider-specific scaffolding)OpenAI55.6%
Gemini 3.1 Pro (Thinking (High); Public split; single attempt; provider-specific scaffolding)Google54.2%
Gemini 3 Pro (Thinking (High); Public split; single attempt; provider-specific scaffolding)Google43.3%

Model reasoning

Intelligence Index v4.215 results

Base model resultArtificial Analysis

Ten-evaluation composite

scoreHigher is better
  1. Claude Fable 5.156.76
  2. GPT-6 Astra54.66
  3. Claude Opus 554.05
  4. Claude Fable 553.19
  5. Muse Spark 1.352.95
  6. GPT-5.6 Sol51.26
  7. Grok 4.650.58
  8. Kimi K350.23
  9. GLM-5.348.58
  10. Gemini 3.8 Flash47.07
  11. Qwen3.8 Max46.91
  12. Muse Spark 1.246.84
  13. GPT-5.6 Terra46.77
  14. Qwen3.8 2.4T A95B46.74
  15. GLM 5.3 Flash46.22
Intelligence Index v4.2 / score
Model and configurationScore
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic56.76
GPT-6 Astra (max)OpenAI54.66
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic54.05
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic53.19
Muse Spark 1.3 (max)Meta52.95
GPT-5.6 Sol (max)OpenAI51.26
Grok 4.6 (high)SpaceXAI50.58
Kimi K3 (max)Kimi50.23
GLM-5.3 (max)Z AI48.58
Gemini 3.8 Flash (high)Google47.07
Qwen3.8 MaxAlibaba46.91
Muse Spark 1.2 (xhigh)Meta46.84
GPT-5.6 Terra (max)OpenAI46.77
Qwen3.8 2.4T A95BAlibaba46.74
GLM 5.3 Flash (GLM-5.3-Flash)Z AI46.22
Humanity's Last Exam15 results

Base model resultArtificial Analysis

Expert-level reasoning and knowledge

%Higher is better
  1. Claude Fable 5.159.13%
  2. Claude Fable 555.47%
  3. Claude Opus 554.87%
  4. GPT-6 Astra54.68%
  5. GPT-5.6 Sol49.49%
  6. Muse Spark 1.349.07%
  7. Gemini 3.7 Flash47.87%
  8. Gemini 3.8 Flash47.82%
  9. Gemini 3.1 Pro Preview47.03%
  10. Kimi K346.9%
  11. Muse Spark 1.245.46%
  12. Grok 4.644.07%
  13. Qwen3.8 Max43.05%
  14. GPT-5.6 Terra42.91%
  15. Grok 4.542.68%
Humanity's Last Exam / %
Model and configurationScore
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic59.13%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic55.47%
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic54.87%
GPT-6 Astra (max)OpenAI54.68%
GPT-5.6 Sol (max)OpenAI49.49%
Muse Spark 1.3 (max)Meta49.07%
Gemini 3.7 Flash (high)Google47.87%
Gemini 3.8 Flash (high)Google47.82%
Gemini 3.1 Pro PreviewGoogle47.03%
Kimi K3 (max)Kimi46.9%
Muse Spark 1.2 (xhigh)Meta45.46%
Grok 4.6 (xhigh)SpaceXAI44.07%
Qwen3.8 MaxAlibaba43.05%
GPT-5.6 Terra (max)OpenAI42.91%
Grok 4.5 (high)SpaceXAI42.68%
GPQA Diamond15 results

Base model resultArtificial Analysis

Graduate-level science

%Higher is better
  1. GPT-6 Astra96.26%
  2. Gemini 3.8 Flash95.25%
  3. Grok 4.694.95%
  4. Gemini 3.7 Flash94.55%
  5. Gemini 3.1 Pro Preview94.14%
  6. Muse Spark 1.394.14%
  7. GPT-5.6 Sol94.14%
  8. Claude Opus 593.74%
  9. Claude Fable 5.193.74%
  10. Qwen3.8 2.4T A95B93.54%
  11. Kimi K393.54%
  12. Grok 4.593.13%
  13. MiniMax-M392.93%
  14. Gemini 3.6 Flash92.83%
  15. DeepSeek V4 Pro 081392.83%
GPQA Diamond / %
Model and configurationScore
GPT-6 Astra (xhigh)OpenAI96.26%
Gemini 3.8 Flash (high)Google95.25%
Grok 4.6 (high)SpaceXAI94.95%
Gemini 3.7 Flash (high)Google94.55%
Gemini 3.1 Pro PreviewGoogle94.14%
Muse Spark 1.3 (xhigh)Meta94.14%
GPT-5.6 Sol (max)OpenAI94.14%
Claude Opus 5 (Adaptive Reasoning, Xhigh Effort)Anthropic93.74%
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic93.74%
Qwen3.8 2.4T A95BAlibaba93.54%
Kimi K3 (max)Kimi93.54%
Grok 4.5 (high)SpaceXAI93.13%
MiniMax-M3MiniMax92.93%
Gemini 3.6 Flash (high)Google92.83%
DeepSeek V4 Pro 0813 (Reasoning, Max Effort)DeepSeek92.83%
CritPt15 results

Base model resultArtificial Analysis

Research-level physics

%Higher is better
  1. GPT-5.6 Sol32.29%
  2. GPT-6 Astra31.71%
  3. Claude Fable 5.131.14%
  4. GPT-5.5 Pro30.57%
  5. GPT-5.6 Terra30%
  6. Claude Opus 529.14%
  7. Claude Fable 528.57%
  8. Muse Spark 1.326%
  9. Gemini 3 Deep Think25.71%
  10. Kimi K323.43%
  11. GLM-5.220.86%
  12. GPT-5.6 Luna20.57%
  13. Qwen3.8 Max20%
  14. Qwen3.8 2.4T A95B20%
  15. Grok 4.619.71%
CritPt / %
Model and configurationScore
GPT-5.6 Sol (max)OpenAI32.29%
GPT-6 Astra (max)OpenAI31.71%
Claude Fable 5.1 (Adaptive Reasoning, Xhigh Effort, Default Fallback)Anthropic31.14%
GPT-5.5 Pro (xhigh)OpenAI30.57%
GPT-5.6 Terra (max)OpenAI30%
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic29.14%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic28.57%
Muse Spark 1.3 (xhigh)Meta26%
Gemini 3 Deep ThinkGoogle25.71%
Kimi K3 (max)Kimi23.43%
GLM-5.2 (max)Z AI20.86%
GPT-5.6 Luna (max)OpenAI20.57%
Qwen3.8 MaxAlibaba20%
Qwen3.8 2.4T A95BAlibaba20%
Grok 4.6 (xhigh)SpaceXAI19.71%
AIME 202515 results

Base model resultArtificial Analysis

Competition mathematics

%Higher is better
  1. Nova 2.0 Lite94.33%
  2. gpt-oss-120b93.44%
  3. NVIDIA Nemotron 3 Nano 30B A3B91%
  4. K-EXAONE90.33%
  5. Nova 2.0 Omni89.67%
  6. gpt-oss-20b89.33%
  7. Nova 2.0 Pro Preview89%
  8. o388.33%
  9. INTELLECT-388%
  10. Apriel-v1.6-15B-Thinker88%
  11. ERNIE 5.0 Thinking Preview85%
  12. Qwen3 Next 80B A3B (Reasoning)84.33%
  13. Ring-flash-2.083.67%
  14. Claude 4.5 Haiku83.67%
  15. Magistral Medium 1.282%
AIME 2025 / %
Model and configurationScore
Nova 2.0 Lite (high)Amazon94.33%
gpt-oss-120b (high)OpenAI93.44%
NVIDIA Nemotron 3 Nano 30B A3B (Reasoning)NVIDIA91%
K-EXAONE (Reasoning)LG AI Research90.33%
Nova 2.0 Omni (medium)Amazon89.67%
gpt-oss-20b (high)OpenAI89.33%
Nova 2.0 Pro Preview (medium)Amazon89%
o3OpenAI88.33%
INTELLECT-3Prime Intellect88%
Apriel-v1.6-15B-ThinkerServiceNow88%
ERNIE 5.0 Thinking PreviewBaidu85%
Qwen3 Next 80B A3B (Reasoning)Alibaba84.33%
Ring-flash-2.0InclusionAI83.67%
Claude 4.5 Haiku (Reasoning)Anthropic83.67%
Magistral Medium 1.2Mistral82%
IFBench15 results

Base model resultArtificial Analysis

Instruction following

%Higher is better
  1. Grok 4.383.33%
  2. MiniMax-M382.86%
  3. Nemotron 3 Ultra 550B A55B81.36%
  4. Nemotron Cascade 2 30B A3B80.41%
  5. MiMo-V2.5-Pro79.86%
  6. Nova 2.0 Pro Preview79.59%
  7. Qwen3.5 397B A17B78.78%
  8. Qwen3.7 Plus77.96%
  9. Gemini 3.1 Pro Preview77.14%
  10. DeepSeek V4 Pro 042476.46%
  11. Qwen3.5 122B A10B75.71%
  12. Gemma 4 31B75.58%
  13. GPT-5.3 Codex75.37%
  14. Gemini 3.5 Flash74.56%
  15. Command A+73.95%
IFBench / %
Model and configurationScore
Grok 4.3 (medium)SpaceXAI83.33%
MiniMax-M3MiniMax82.86%
Nemotron 3 Ultra 550B A55B (Reasoning)NVIDIA81.36%
Nemotron Cascade 2 30B A3BNVIDIA80.41%
MiMo-V2.5-ProXiaomi79.86%
Nova 2.0 Pro Preview (low)Amazon79.59%
Qwen3.5 397B A17B (Reasoning)Alibaba78.78%
Qwen3.7 PlusAlibaba77.96%
Gemini 3.1 Pro PreviewGoogle77.14%
DeepSeek V4 Pro 0424 (DeepSeek V4 Pro (Reasoning, Max Effort))DeepSeek76.46%
Qwen3.5 122B A10B (Reasoning)Alibaba75.71%
Gemma 4 31B (Reasoning)Google75.58%
GPT-5.3 Codex (xhigh)OpenAI75.37%
Gemini 3.5 Flash (medium)Google74.56%
Command A+Cohere73.95%
LongBench v2 / Moonshot report5 results

Vendor-reported model resultMoonshot AI (model-card report)

Reading and reasoning over long inputs.

scoreHigher is better
  1. Gemini 3 Pro68.2
  2. Claude Opus 4.564.4
  3. Kimi K2.561
  4. DeepSeek V3.259.8
  5. GPT-5.254.5
LongBench v2 / Moonshot report / score
Model and configurationScore
Gemini 3 Pro (High Thinking Level; publisher rerun (*))Google68.2
Claude Opus 4.5 (Extended Thinking; publisher rerun (*))Anthropic64.4
Kimi K2.5 (Thinking)Moonshot AI61
DeepSeek V3.2 (Thinking; publisher rerun (*))DeepSeek59.8
GPT-5.2 (xhigh; publisher rerun (*))OpenAI54.5
AIME 2026 I / Z.ai report6 results

Vendor-reported model resultZ.ai (model-card report)

The first 2026 AIME exam, not the combined 2026 set.

scoreHigher is better
  1. Claude Opus 4.593.3
  2. GLM-4.792.9
  3. GLM-592.7
  4. DeepSeek V3.292.7
  5. Kimi K2.592.5
  6. Gemini 3 Pro90.6
AIME 2026 I / Z.ai report / score
Model and configurationScore
Claude Opus 4.5 (GLM-5 release comparison)Anthropic93.3
GLM-4.7 (GLM-5 release comparison)Z.ai92.9
GLM-5 release reportZ.ai92.7
DeepSeek V3.2 (GLM-5 release comparison)DeepSeek92.7
Kimi K2.5 (GLM-5 release comparison)Moonshot AI92.5
Gemini 3 Pro (GLM-5 release comparison)Google90.6
HMMT November 2025 / Z.ai report7 results

Vendor-reported model resultZ.ai (model-card report)

The November 2025 mathematics competition.

scoreHigher is better
  1. GPT-5.297.1
  2. GLM-596.9
  3. GLM-4.793.5
  4. Gemini 3 Pro93
  5. Claude Opus 4.591.7
  6. Kimi K2.591.1
  7. DeepSeek V3.290.2
HMMT November 2025 / Z.ai report / score
Model and configurationScore
GPT-5.2 (xhigh)OpenAI97.1
GLM-5 release reportZ.ai96.9
GLM-4.7 (GLM-5 release comparison)Z.ai93.5
Gemini 3 Pro (GLM-5 release comparison)Google93
Claude Opus 4.5 (GLM-5 release comparison)Anthropic91.7
Kimi K2.5 (GLM-5 release comparison)Moonshot AI91.1
DeepSeek V3.2 (GLM-5 release comparison)DeepSeek90.2
AA-LCR / MiniMax report7 results

Vendor-reported model resultMiniMax (internal evaluation)

Reasoning across long inputs.

scoreHigher is better
  1. Claude Opus 4.574
  2. GPT-5.273
  3. Claude Opus 4.671
  4. Gemini 3 Pro71
  5. MiniMax-M2.569.5
  6. Claude Sonnet 4.566
  7. MiniMax-M2.162
AA-LCR / MiniMax report / score
Model and configurationScore
Claude Opus 4.5 (MiniMax internal evaluation)Anthropic74
GPT-5.2 (Thinking; MiniMax internal evaluation)OpenAI73
Claude Opus 4.6 (MiniMax internal evaluation)Anthropic71
Gemini 3 Pro (MiniMax internal evaluation)Google71
MiniMax-M2.5 (MiniMax internal evaluation)MiniMax69.5
Claude Sonnet 4.5 (MiniMax internal evaluation)Anthropic66
MiniMax-M2.1 (MiniMax internal evaluation)MiniMax62
HLE no tools / DeepSeek GA report8 results

Vendor-reported model resultDeepSeek (GA model-card report)

Reasoning across disciplines without tools.

scoreHigher is better
  1. Fable-5 (w/ fallback)53.3
  2. Opus-4.849.8
  3. Kimi K343.5
  4. DeepSeek-V4-Pro-081342.7
  5. GLM-5.240.5
  6. DeepSeek-V4-Flash-073137.8
  7. DeepSeek-V4-Pro (Preview)37.7
  8. DeepSeek-V4-Flash (Preview)34.8
HLE no tools / DeepSeek GA report / score
Model and configurationScore
Fable-5 (w/ fallback) (No tools; HLE reasoning effort and token budget not disclosed; with fallback, details unspecified)Anthropic53.3
Opus-4.8 (No tools; HLE reasoning effort and token budget not disclosed)Anthropic49.8
Kimi K3 (No tools; HLE reasoning effort and token budget not disclosed)Moonshot AI43.5
DeepSeek-V4-Pro-0813 (No tools; HLE reasoning effort and token budget not disclosed)DeepSeek42.7
GLM-5.2 (No tools; HLE reasoning effort and token budget not disclosed)Z.ai40.5
DeepSeek-V4-Flash-0731 (No tools; HLE reasoning effort and token budget not disclosed)DeepSeek37.8
DeepSeek-V4-Pro (Preview) (No tools; HLE reasoning effort and token budget not disclosed)DeepSeek37.7
DeepSeek-V4-Flash (Preview) (No tools; HLE reasoning effort and token budget not disclosed)DeepSeek34.8
HLE with tools / DeepSeek GA report8 results

Vendor-reported model resultDeepSeek (GA model-card report)

Reasoning across disciplines with tools.

scoreHigher is better
  1. Fable-5 (w/ fallback)63
  2. DeepSeek-V4-Pro-081360
  3. Opus-4.857.9
  4. Kimi K356
  5. GLM-5.254.7
  6. DeepSeek-V4-Flash-073151.5
  7. DeepSeek-V4-Pro (Preview)48.2
  8. DeepSeek-V4-Flash (Preview)45.1
HLE with tools / DeepSeek GA report / score
Model and configurationScore
Fable-5 (w/ fallback) (With tools; specific tools, HLE reasoning effort and token budget not disclosed; with fallback, details unspecified)Anthropic63
DeepSeek-V4-Pro-0813 (With tools; specific tools, HLE reasoning effort and token budget not disclosed)DeepSeek60
Opus-4.8 (With tools; specific tools, HLE reasoning effort and token budget not disclosed)Anthropic57.9
Kimi K3 (With tools; specific tools, HLE reasoning effort and token budget not disclosed)Moonshot AI56
GLM-5.2 (With tools; specific tools, HLE reasoning effort and token budget not disclosed)Z.ai54.7
DeepSeek-V4-Flash-0731 (With tools; specific tools, HLE reasoning effort and token budget not disclosed)DeepSeek51.5
DeepSeek-V4-Pro (Preview) (With tools; specific tools, HLE reasoning effort and token budget not disclosed)DeepSeek48.2
DeepSeek-V4-Flash (Preview) (With tools; specific tools, HLE reasoning effort and token budget not disclosed)DeepSeek45.1
HMMT February 2026 / DeepSeek Preview report6 results

Vendor-reported model resultDeepSeek (Preview technical report)

Mathematics competition problems, measured by Pass@1.

%Higher is better
  1. GPT-5.4 xHigh97.7%
  2. Opus-4.6 Max96.2%
  3. DS-V4-Pro Max95.2%
  4. Gemini-3.1-Pro High94.7%
  5. K2.6 Thinking92.7%
  6. GLM-5.1 Thinking89.4%
HMMT February 2026 / DeepSeek Preview report / %
Model and configurationScore
GPT-5.4 xHigh (xHigh; DeepSeek Preview report comparison)OpenAI97.7%
Opus-4.6 Max (Max; DeepSeek Preview report comparison)Anthropic96.2%
DS-V4-Pro Max (Preview; Max; temperature=1.0; 384K-token context; distinct rigorous-proof math prompt)DeepSeek95.2%
Gemini-3.1-Pro High (High; DeepSeek Preview report comparison)Google94.7%
K2.6 Thinking (Thinking; DeepSeek Preview report comparison)Moonshot AI92.7%
GLM-5.1 Thinking (Thinking; DeepSeek Preview report comparison)Z.ai89.4%
LongBench v2 / Qwen3.8-Max report4 results

Vendor-reported model resultQwen (model-card report)

Reading and reasoning over long documents and inputs.

scoreHigher is better
  1. Opus 4.869.1
  2. GPT 5.6 Sol (max)67.1
  3. Qwen3.8-Max66.3
  4. Qwen3.7-Max65.3
LongBench v2 / Qwen3.8-Max report / score
Model and configurationScore
Opus 4.8 (LongBench v2; tools, reasoning effort and token budget not disclosed)Anthropic69.1
GPT 5.6 Sol (max) (max (table header); LongBench v2 tools and token budget not disclosed)OpenAI67.1
Qwen3.8-Max service model; LongBench v2 tools, reasoning effort and token budget not disclosedQwen66.3
Qwen3.7-Max (LongBench v2; tools, reasoning effort and token budget not disclosed)Qwen65.3
GPQA Diamond / Google G31 (2026-02-19)5 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.1 Pro release comparison)

Scientific knowledge and reasoning without tools.

%Higher is better
  1. Gemini 3.1 Pro94.3%
  2. GPT-5.292.4%
  3. Gemini 3 Pro91.9%
  4. Claude Opus 4.691.3%
  5. Claude Sonnet 4.689.9%
GPQA Diamond / Google G31 (2026-02-19) / %
Model and configurationScore
Gemini 3.1 Pro (Thinking (High); No tools)Google94.3%
GPT-5.2 (Thinking (xhigh); No tools)OpenAI92.4%
Gemini 3 Pro (Thinking (High); No tools)Google91.9%
Claude Opus 4.6 (Thinking (Max); No tools)Anthropic91.3%
Claude Sonnet 4.6 (Thinking (Max); No tools)Anthropic89.9%
HLE, no tools / Google G31 (2026-02-19)5 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.1 Pro release comparison)

Expert-level reasoning on the full text and multimodal HLE set without tools.

%Higher is better
  1. Gemini 3.1 Pro44.4%
  2. Claude Opus 4.640%
  3. Gemini 3 Pro37.5%
  4. GPT-5.234.5%
  5. Claude Sonnet 4.633.2%
HLE, no tools / Google G31 (2026-02-19) / %
Model and configurationScore
Gemini 3.1 Pro (Thinking (High); No tools)Google44.4%
Claude Opus 4.6 (Thinking (Max); No tools)Anthropic40%
Gemini 3 Pro (Thinking (High); No tools)Google37.5%
GPT-5.2 (Thinking (xhigh); No tools)OpenAI34.5%
Claude Sonnet 4.6 (Thinking (Max); No tools)Anthropic33.2%
HLE, search and code / Google G31 (2026-02-19)5 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.1 Pro release comparison)

Expert-level HLE reasoning with search and code execution.

%Higher is better
  1. Claude Opus 4.653.1%
  2. Gemini 3.1 Pro51.4%
  3. Claude Sonnet 4.649%
  4. Gemini 3 Pro45.8%
  5. GPT-5.245.5%
HLE, search and code / Google G31 (2026-02-19) / %
Model and configurationScore
Claude Opus 4.6 (Thinking (Max); Search and code; provider-specific tools)Anthropic53.1%
Gemini 3.1 Pro (Thinking (High); Search and code; provider-specific tools)Google51.4%
Claude Sonnet 4.6 (Thinking (Max); Search and code; provider-specific tools)Anthropic49%
Gemini 3 Pro (Thinking (High); Search and code; provider-specific tools)Google45.8%
GPT-5.2 (Thinking (xhigh); Search and code; provider-specific tools)OpenAI45.5%
ARC-AGI-2 / Google G31 (2026-02-19)5 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.1 Pro release comparison)

Inferring unfamiliar patterns from example grids.

%Higher is better
  1. Gemini 3.1 Pro77.1%
  2. Claude Opus 4.668.8%
  3. Claude Sonnet 4.658.3%
  4. GPT-5.252.9%
  5. Gemini 3 Pro31.1%
ARC-AGI-2 / Google G31 (2026-02-19) / %
Model and configurationScore
Gemini 3.1 Pro (Thinking (High); ARC Prize Verified; semi-private set)Google77.1%
Claude Opus 4.6 (Thinking (Max); ARC Prize Verified; semi-private set)Anthropic68.8%
Claude Sonnet 4.6 (Thinking (Max); ARC Prize Verified; semi-private set)Anthropic58.3%
GPT-5.2 (Thinking (xhigh); ARC Prize Verified; semi-private set)OpenAI52.9%
Gemini 3 Pro (Thinking (High); ARC Prize Verified; semi-private set)Google31.1%
AIME 2025, no tools / Google G3F (2025-12-17)7 results

Vendor-reported model resultGoogle DeepMind (Gemini 3 Flash release comparison)

Mathematics from the 2025 AIME competition.

%Higher is better
  1. GPT-5.2100%
  2. Gemini 3 Flash95.2%
  3. Gemini 3 Pro95%
  4. Grok 4.1 Fast91.9%
  5. Gemini 2.5 Pro88%
  6. Claude Sonnet 4.587%
  7. Gemini 2.5 Flash72%
AIME 2025, no tools / Google G3F (2025-12-17) / %
Model and configurationScore
GPT-5.2 (Extra high; No tools)OpenAI100%
Gemini 3 Flash (Thinking; exact level unspecified; No tools)Google95.2%
Gemini 3 Pro (Thinking; exact level unspecified; No tools)Google95%
Grok 4.1 Fast (Reasoning; No tools)xAI91.9%
Gemini 2.5 Pro (Thinking; exact level unspecified; No tools)Google88%
Claude Sonnet 4.5 (Thinking; high preferred, otherwise best available reported setting; No tools)Anthropic87%
Gemini 2.5 Flash (Thinking; exact level unspecified; No tools)Google72%
AIME 2025, code execution / Google G3F (2025-12-17)4 results

Vendor-reported model resultGoogle DeepMind (Gemini 3 Flash release comparison)

Mathematics from the 2025 AIME competition.

%Higher is better
  1. Gemini 3 Pro100%
  2. Claude Sonnet 4.5100%
  3. Gemini 3 Flash99.7%
  4. Gemini 2.5 Flash75.7%
AIME 2025, code execution / Google G3F (2025-12-17) / %
Model and configurationScore
Gemini 3 Pro (Thinking; exact level unspecified; Code execution)Google100%
Claude Sonnet 4.5 (Thinking; high preferred, otherwise best available reported setting; Code execution)Anthropic100%
Gemini 3 Flash (Thinking; exact level unspecified; Code execution)Google99.7%
Gemini 2.5 Flash (Thinking; exact level unspecified; Code execution)Google75.7%
HLE, no tools / Anthropic A51 (2026-09-01)3 results

Vendor-reported model resultAnthropic (Fable 5.1 release comparison)

Original HLE reasoning on 2,500 multimodal questions without tools.

%Higher is better
  1. Claude Fable 5.160.9%
  2. Claude Fable 557.8%
  3. Claude Opus 556.6%
HLE, no tools / Anthropic A51 (2026-09-01) / %
Model and configurationScore
Claude Fable 5.1 (Adaptive thinking (auto); max effort; default sampling; five-trial mean; total token cap 1M; no compaction; production safeguards enabled; No tools)Anthropic60.9%
Claude Fable 5 (Detailed reasoning setting and budget unverified; production safeguards enabled; No tools)Anthropic57.8%
Claude Opus 5 (Detailed reasoning setting and budget unverified; No tools)Anthropic56.6%
HLE, tools / Anthropic A51 (2026-09-01)3 results

Vendor-reported model resultAnthropic (Fable 5.1 release comparison)

Original HLE reasoning with search, web retrieval and code tools.

%Higher is better
  1. Claude Fable 5.165%
  2. Claude Fable 563.8%
  3. Claude Opus 563.6%
HLE, tools / Anthropic A51 (2026-09-01) / %
Model and configurationScore
Claude Fable 5.1 (Adaptive thinking (auto); max effort; default sampling; five-trial mean; total token cap 1M; no compaction; production safeguards enabled; Search; restricted fetch; programmatic tool calling; code execution)Anthropic65%
Claude Fable 5 (Detailed reasoning setting and budget unverified; production safeguards enabled; Search; restricted fetch; programmatic tool calling; code execution)Anthropic63.8%
Claude Opus 5 (Detailed reasoning setting and budget unverified; Search; restricted fetch; programmatic tool calling; code execution)Anthropic63.6%
ARC-AGI-2 / Anthropic A51 (2026-09-01)4 results

Vendor-reported model resultAnthropic (Fable 5.1 release comparison)

Novel grid-pattern reasoning in the September release comparison.

%Higher is better
  1. GPT-5.6 Sol92.5%
  2. Claude Opus 590.42%
  3. Claude Fable 5.1 / Claude Mythos 5.190%
  4. Claude Fable 5 / Claude Mythos 589.2%
ARC-AGI-2 / Anthropic A51 (2026-09-01) / %
Model and configurationScore
GPT-5.6 Sol (Exact reasoning setting and token budget unverified)OpenAI92.5%
Claude Opus 5 (Exact reasoning setting and token budget unverified)Anthropic90.42%
Claude Fable 5.1 / Claude Mythos 5.1 (Fable 5.1: max effort; semi-private set; ARC Prize Verified)Anthropic90%
Claude Fable 5 / Claude Mythos 5 (Exact reasoning setting and token budget unverified)Anthropic89.2%
HLE-Verified / Google G38 (2026-09-02)6 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.8 Flash release comparison)

Expert reasoning on a verified and revised 1,811-question set.

%Higher is better
  1. Gemini 3.8 Flash54.9%
  2. GPT-5.6 Sol54.5%
  3. Claude Opus 554.4%
  4. Gemini 3.7 Flash53.6%
  5. GPT-5.6 Terra51.1%
  6. Claude Sonnet 531%
HLE-Verified / Google G38 (2026-09-02) / %
Model and configurationScore
Gemini 3.8 Flash (Default sampling; exact thinking level unspecified; HLE-Verified; 1811 items; tool use unspecified)Google54.9%
GPT-5.6 Sol (Exact reasoning level unspecified; HLE-Verified; 1811 items; tool use unspecified)OpenAI54.5%
Claude Opus 5 (Exact thinking level unspecified; HLE-Verified; 1811 items; tool use unspecified)Anthropic54.4%
Gemini 3.7 Flash (Exact thinking level unspecified; HLE-Verified; 1811 items; tool use unspecified)Google53.6%
GPT-5.6 Terra (Maximum reasoning preferred; otherwise best available reported setting; HLE-Verified; 1811 items; tool use unspecified)OpenAI51.1%
Claude Sonnet 5 (Maximum thinking preferred; otherwise best available reported setting; HLE-Verified; 1811 items; tool use unspecified)Anthropic31%

Documents and context

AA-LCR v1.115 results

Base model resultArtificial Analysis

Long-context reasoning

%Higher is better
  1. Kimi K388.67%
  2. Claude Fable 5.185.33%
  3. Muse Spark 1.384.33%
  4. Gemini 3.8 Flash84%
  5. GPT-5.6 Sol84%
  6. GPT-5.6 Luna83.67%
  7. Muse Glimmer83.33%
  8. GPT-5.3 Codex83.33%
  9. MiniMax-M383%
  10. GPT-5.6 Terra83%
  11. Gemini 3.7 Flash83%
  12. Agnes 2.5 Pro Beta83%
  13. Claude Fable 582.33%
  14. Claude Sonnet 582%
  15. Qwen3.8 27B82%
AA-LCR v1.1 / %
Model and configurationScore
Kimi K3 (max)Kimi88.67%
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic85.33%
Muse Spark 1.3 (max)Meta84.33%
Gemini 3.8 Flash (medium)Google84%
GPT-5.6 Sol (max)OpenAI84%
GPT-5.6 Luna (max)OpenAI83.67%
Muse Glimmer (high)Meta83.33%
GPT-5.3 Codex (xhigh)OpenAI83.33%
MiniMax-M3MiniMax83%
GPT-5.6 Terra (max)OpenAI83%
Gemini 3.7 Flash (medium)Google83%
Agnes 2.5 Pro BetaSapiens AI83%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic82.33%
Claude Sonnet 5 (Adaptive Reasoning, Max Effort)Anthropic82%
Qwen3.8 27B (xhigh)Alibaba82%
MMMU-Pro15 results

Base model resultArtificial Analysis

Visual reasoning

%Higher is better
  1. GPT-6 Astra86.88%
  2. Gemini 3.8 Flash85.61%
  3. Gemini 3.7 Flash85.49%
  4. Claude Opus 584.74%
  5. Gemini 3.5 Flash83.87%
  6. GPT-5.6 Sol83.41%
  7. Gemini 3.6 Flash83.24%
  8. Gemini 3.1 Pro Preview82.43%
  9. Qwen3.8 Max82.31%
  10. Muse Spark 1.382.02%
  11. GPT-5.6 Terra80.69%
  12. Kimi K380.52%
  13. Qwen3.7 Plus80.46%
  14. Grok 4.580.4%
  15. Qwen3.8-Flash-Next79.77%
MMMU-Pro / %
Model and configurationScore
GPT-6 Astra (max)OpenAI86.88%
Gemini 3.8 Flash (high)Google85.61%
Gemini 3.7 Flash (high)Google85.49%
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic84.74%
Gemini 3.5 Flash (medium)Google83.87%
GPT-5.6 Sol (max)OpenAI83.41%
Gemini 3.6 Flash (high)Google83.24%
Gemini 3.1 Pro PreviewGoogle82.43%
Qwen3.8 MaxAlibaba82.31%
Muse Spark 1.3 (xhigh)Meta82.02%
GPT-5.6 Terra (max)OpenAI80.69%
Kimi K3 (max)Kimi80.52%
Qwen3.7 PlusAlibaba80.46%
Grok 4.5 (high)SpaceXAI80.4%
Qwen3.8-Flash-NextAlibaba79.77%
AA-Omniscience / Index15 results

Base model resultArtificial Analysis

Knowledge reliability

scoreHigher is better
  1. GPT-6 Astra43.73
  2. Claude Fable 5.143.45
  3. Claude Fable 543.3
  4. Claude Opus 537.07
  5. Gemini 3.1 Pro Preview31.88
  6. Grok 4.630.48
  7. Gemini 3.8 Flash29.55
  8. Muse Spark 1.227.2
  9. Gemini 3.7 Flash26.48
  10. Grok 4.525.32
  11. Muse Spark 1.324.93
  12. Gemini 3.6 Flash22.13
  13. GPT-5.6 Sol21.97
  14. Gemini 3.5 Flash20.82
  15. Kimi K319.7
AA-Omniscience / Index / score
Model and configurationScore
GPT-6 Astra (high)OpenAI43.73
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic43.45
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic43.3
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic37.07
Gemini 3.1 Pro PreviewGoogle31.88
Grok 4.6 (high)SpaceXAI30.48
Gemini 3.8 Flash (high)Google29.55
Muse Spark 1.2 (xhigh)Meta27.2
Gemini 3.7 Flash (high)Google26.48
Grok 4.5 (high)SpaceXAI25.32
Muse Spark 1.3 (max)Meta24.93
Gemini 3.6 Flash (high)Google22.13
GPT-5.6 Sol (max)OpenAI21.97
Gemini 3.5 Flash (medium)Google20.82
Kimi K3 (max)Kimi19.7
AA-Omniscience / Accuracy15 results

Base model resultArtificial Analysis

Correct knowledge answers

%Higher is better
  1. Claude Fable 5.167.23%
  2. Claude Fable 565.35%
  3. GPT-6 Astra62.6%
  4. Claude Opus 560.87%
  5. GPT-5.6 Sol59.4%
  6. Gemini 3.7 Flash55.32%
  7. Gemini 3.1 Pro Preview54.85%
  8. Gemini 3.8 Flash54.6%
  9. GPT-5.3 Codex52.88%
  10. Grok 4.551.55%
  11. Gemini 3.5 Flash51.05%
  12. Gemini 3.6 Flash49.97%
  13. DeepSeek V4 Pro 081349.1%
  14. Grok 4.648.23%
  15. Kimi K347.58%
AA-Omniscience / Accuracy / %
Model and configurationScore
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic67.23%
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic65.35%
GPT-6 Astra (max)OpenAI62.6%
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic60.87%
GPT-5.6 Sol (max)OpenAI59.4%
Gemini 3.7 Flash (high)Google55.32%
Gemini 3.1 Pro PreviewGoogle54.85%
Gemini 3.8 Flash (high)Google54.6%
GPT-5.3 Codex (xhigh)OpenAI52.88%
Grok 4.5 (high)SpaceXAI51.55%
Gemini 3.5 Flash (medium)Google51.05%
Gemini 3.6 Flash (high)Google49.97%
DeepSeek V4 Pro 0813 (Reasoning, Max Effort)DeepSeek49.1%
Grok 4.6 (high)SpaceXAI48.23%
Kimi K3 (max)Kimi47.58%
AA-Omniscience / Hallucination15 results

Base model resultArtificial Analysis

Hallucination rate, lower is better

%Lower is better
  1. MiniCPM5-1B0.9%
  2. G9v3-3B11.66%
  3. G9v3-39A5B13.04%
  4. Command A+14.16%
  5. LFM2.5-2.6B16%
  6. Grok 4.316.94%
  7. Qwen3.8 27B18.09%
  8. MiniMax-M318.43%
  9. Quasar 438B21.44%
  10. K-EXAONE 2.0 080322.6%
  11. Grok 4.624%
  12. Solar Pro 424.4%
  13. MiMo-V2.5-Pro24.7%
  14. Solar Open2 250B25.38%
  15. Granite 4.2 30B25.58%
AA-Omniscience / Hallucination / %
Model and configurationScore
MiniCPM5-1B (Non-reasoning)OpenBMB0.9%
G9v3-3BAI9Stars11.66%
G9v3-39A5BAI9Stars13.04%
Command A+Cohere14.16%
LFM2.5-2.6BLiquid AI16%
Grok 4.3 (medium)SpaceXAI16.94%
Qwen3.8 27B (Non-reasoning)Alibaba18.09%
MiniMax-M3MiniMax18.43%
Quasar 438B (max, based on GLM-5.2)Multiverse Computing21.44%
K-EXAONE 2.0 0803 (K-EXAONE 2.0)LG AI Research22.6%
Grok 4.6 (medium)SpaceXAI24%
Solar Pro 4Upstage24.4%
MiMo-V2.5-ProXiaomi24.7%
Solar Open2 250BUpstage25.38%
Granite 4.2 30BIBM25.58%
MathVision / Moonshot report5 results

Vendor-reported model resultMoonshot AI (model-card report)

Mathematics from visual problems.

scoreHigher is better
  1. Gemini 3 Pro86.1
  2. Kimi K2.584.2
  3. GPT-5.283
  4. Claude Opus 4.577.1
  5. Qwen3-VL-235B-A22B74.6
MathVision / Moonshot report / score
Model and configurationScore
Gemini 3 Pro (High Thinking Level; publisher rerun (*))Google86.1
Kimi K2.5 (Thinking)Moonshot AI84.2
GPT-5.2 (xhigh)OpenAI83
Claude Opus 4.5 (Extended Thinking; publisher rerun (*))Anthropic77.1
Qwen3-VL-235B-A22B (Thinking)Qwen74.6
OCRBench / Moonshot report5 results

Vendor-reported model resultMoonshot AI (model-card report)

Text recognition in images.

scoreHigher is better
  1. Kimi K2.592.3
  2. Gemini 3 Pro90.3
  3. Qwen3-VL-235B-A22B87.5
  4. Claude Opus 4.586.5
  5. GPT-5.280.7
OCRBench / Moonshot report / score
Model and configurationScore
Kimi K2.5 (Thinking)Moonshot AI92.3
Gemini 3 Pro (High Thinking Level; publisher rerun (*))Google90.3
Qwen3-VL-235B-A22B (Thinking)Qwen87.5
Claude Opus 4.5 (Extended Thinking; publisher rerun (*))Anthropic86.5
GPT-5.2 (xhigh; publisher rerun (*))OpenAI80.7
OmniDocBench 1.5 / Moonshot report5 results

Vendor-reported model resultMoonshot AI (model-card report)

Document reading. The reported score is (1 − normalized edit distance) × 100.

scoreHigher is better
  1. Kimi K2.588.8
  2. Gemini 3 Pro88.5
  3. Claude Opus 4.587.7
  4. GPT-5.285.7
  5. Qwen3-VL-235B-A22B82
OmniDocBench 1.5 / Moonshot report / score
Model and configurationScore
Kimi K2.5 (Thinking)Moonshot AI88.8
Gemini 3 Pro (High Thinking Level)Google88.5
Claude Opus 4.5 (Extended Thinking; publisher rerun (*))Anthropic87.7
GPT-5.2 (xhigh)OpenAI85.7
Qwen3-VL-235B-A22B (Thinking; publisher rerun (*))Qwen82
VideoMMMU / Moonshot report5 results

Vendor-reported model resultMoonshot AI (model-card report)

Video understanding across subjects.

scoreHigher is better
  1. Gemini 3 Pro87.6
  2. Kimi K2.586.6
  3. GPT-5.285.9
  4. Claude Opus 4.584.4
  5. Qwen3-VL-235B-A22B80
VideoMMMU / Moonshot report / score
Model and configurationScore
Gemini 3 Pro (High Thinking Level)Google87.6
Kimi K2.5 (Thinking)Moonshot AI86.6
GPT-5.2 (xhigh)OpenAI85.9
Claude Opus 4.5 (Extended Thinking; publisher rerun (*))Anthropic84.4
Qwen3-VL-235B-A22B (Thinking)Qwen80
MotionBench / Moonshot report4 results

Vendor-reported model resultMoonshot AI (model-card report)

Understanding motion in video.

scoreHigher is better
  1. Kimi K2.570.4
  2. Gemini 3 Pro70.3
  3. GPT-5.264.8
  4. Claude Opus 4.560.3
MotionBench / Moonshot report / score
Model and configurationScore
Kimi K2.5 (Thinking)Moonshot AI70.4
Gemini 3 Pro (High Thinking Level)Google70.3
GPT-5.2 (xhigh)OpenAI64.8
Claude Opus 4.5 (Extended Thinking)Anthropic60.3
IFBench / MiniMax report7 results

Vendor-reported model resultMiniMax (internal evaluation)

Following detailed instructions.

scoreHigher is better
  1. GPT-5.275
  2. MiniMax-M2.570
  3. MiniMax-M2.170
  4. Gemini 3 Pro70
  5. Claude Opus 4.558
  6. Claude Sonnet 4.557
  7. Claude Opus 4.653
IFBench / MiniMax report / score
Model and configurationScore
GPT-5.2 (Thinking; MiniMax internal evaluation)OpenAI75
MiniMax-M2.5 (MiniMax internal evaluation)MiniMax70
MiniMax-M2.1 (MiniMax internal evaluation)MiniMax70
Gemini 3 Pro (MiniMax internal evaluation)Google70
Claude Opus 4.5 (MiniMax internal evaluation)Anthropic58
Claude Sonnet 4.5 (MiniMax internal evaluation)Anthropic57
Claude Opus 4.6 (MiniMax internal evaluation)Anthropic53
MMMLU / DeepSeek Base report3 results

Vendor-reported model resultDeepSeek (Base technical report)

Multilingual knowledge, measured by exact match with five examples.

%Higher is better
  1. DeepSeek-V4-Pro-Base90.3%
  2. DeepSeek-V4-Flash-Base88.8%
  3. DeepSeek-V3.2-Base87.9%
MMMLU / DeepSeek Base report / %
Model and configurationScore
DeepSeek-V4-Pro-Base (Base; MMMLU EM; 5-shot; shared internal evaluation setup; sampling and tools not disclosed)DeepSeek90.3%
DeepSeek-V4-Flash-Base (Base; MMMLU EM; 5-shot; shared internal evaluation setup; sampling and tools not disclosed)DeepSeek88.8%
DeepSeek-V3.2-Base (Base; MMMLU EM; 5-shot; shared internal evaluation setup; sampling and tools not disclosed)DeepSeek87.9%
OmniDocBench 1.5 / Qwen3.8-27B report5 results

Vendor-reported model resultQwen (model-card report)

Reading document images, using the publisher's reported score.

scoreHigher is better
  1. Qwen3.7-Plus91.4
  2. Qwen3.8-27B91.1
  3. Qwen3.6-27B89.4
  4. Opus4.6 Max86.6
  5. Muse Glimmer-30B75.8
OmniDocBench 1.5 / Qwen3.8-27B report / score
Model and configurationScore
Qwen3.7-Plus (OmniDocBench 1.5; tools, reasoning effort and image settings not disclosed)Qwen91.4
Qwen3.8-27B (OmniDocBench 1.5; tools, reasoning effort and image settings not disclosed)Qwen91.1
Qwen3.6-27B (OmniDocBench 1.5; tools, reasoning effort and image settings not disclosed)Qwen89.4
Opus4.6 Max (Max (table header); OmniDocBench 1.5 tools and image settings not disclosed)Anthropic86.6
Muse Glimmer-30B (OmniDocBench 1.5; tools, reasoning effort and image settings not disclosed)Meta75.8
MMMU-Pro / Google G31 (2026-02-19)5 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.1 Pro release comparison)

Multimodal understanding and reasoning without tools.

%Higher is better
  1. Gemini 3 Pro81%
  2. Gemini 3.1 Pro80.5%
  3. GPT-5.279.5%
  4. Claude Sonnet 4.674.5%
  5. Claude Opus 4.673.9%
MMMU-Pro / Google G31 (2026-02-19) / %
Model and configurationScore
Gemini 3 Pro (Thinking (High); No tools; Standard (10 options) and Vision average)Google81%
Gemini 3.1 Pro (Thinking (High); No tools; Standard (10 options) and Vision average)Google80.5%
GPT-5.2 (Thinking (xhigh); No tools; Standard (10 options) and Vision average)OpenAI79.5%
Claude Sonnet 4.6 (Thinking (Max); No tools; Standard (10 options) and Vision average)Anthropic74.5%
Claude Opus 4.6 (Thinking (Max); No tools; Standard (10 options) and Vision average)Anthropic73.9%
MRCR v2, 128k / Google G31 (2026-02-19)5 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.1 Pro release comparison)

Retrieving eight target items from long context, averaged up to 128k tokens.

%Higher is better
  1. Gemini 3.1 Pro84.9%
  2. Claude Sonnet 4.684.9%
  3. Claude Opus 4.684%
  4. GPT-5.283.8%
  5. Gemini 3 Pro77%
MRCR v2, 128k / Google G31 (2026-02-19) / %
Model and configurationScore
Gemini 3.1 Pro (Thinking (High); MRCR v2; 8 needles; 128k cumulative average)Google84.9%
Claude Sonnet 4.6 (Thinking (Max); MRCR v2; 8 needles; 128k cumulative average)Anthropic84.9%
Claude Opus 4.6 (Thinking (Max); MRCR v2; 8 needles; 128k cumulative average)Anthropic84%
GPT-5.2 (Thinking (xhigh); MRCR v2; 8 needles; 128k cumulative average)OpenAI83.8%
Gemini 3 Pro (Thinking (High); MRCR v2; 8 needles; 128k cumulative average)Google77%
MRCR v2, 1M / Google G31 (2026-02-19)2 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.1 Pro release comparison)

Retrieving eight target items at the 1M-token context point.

%Higher is better
  1. Gemini 3.1 Pro26.3%
  2. Gemini 3 Pro26.3%
MRCR v2, 1M / Google G31 (2026-02-19) / %
Model and configurationScore
Gemini 3.1 Pro (Thinking (High); MRCR v2; 8 needles; 1M pointwise)Google26.3%
Gemini 3 Pro (Thinking (High); MRCR v2; 8 needles; 1M pointwise)Google26.3%
CharXiv Reasoning / Google G38 (2026-09-02)6 results

Vendor-reported model resultGoogle DeepMind (Gemini 3.8 Flash release comparison)

Combining information from complex charts without tools.

%Higher is better
  1. Gemini 3.8 Flash86.2%
  2. GPT-5.6 Terra85.9%
  3. GPT-5.6 Sol85.8%
  4. Gemini 3.7 Flash84.5%
  5. Claude Opus 583.7%
  6. Claude Sonnet 570.1%
CharXiv Reasoning / Google G38 (2026-09-02) / %
Model and configurationScore
Gemini 3.8 Flash (Default sampling; exact thinking level unspecified; No tools)Google86.2%
GPT-5.6 Terra (Maximum reasoning preferred; otherwise best available reported setting; No tools)OpenAI85.9%
GPT-5.6 Sol (Exact reasoning level unspecified; No tools)OpenAI85.8%
Gemini 3.7 Flash (Exact thinking level unspecified; No tools)Google84.5%
Claude Opus 5 (Exact thinking level unspecified; No tools)Anthropic83.7%
Claude Sonnet 5 (Maximum thinking preferred; otherwise best available reported setting; No tools)Anthropic70.1%

Retrieval models

MTEB / OpenAI embeddings report3 results

Retrieval model resultOpenAI (developer documentation)

Text embedding evaluation, separate from generative model reasoning.

%Higher is better
  1. text-embedding-3-large64.6%
  2. text-embedding-3-small62.3%
  3. text-embedding-ada-00261%
MTEB / OpenAI embeddings report / %
Model and configurationScore
text-embedding-3-large (Published MTEB result; evaluation dimensions unspecified)OpenAI64.6%
text-embedding-3-small (Published MTEB result; evaluation dimensions unspecified)OpenAI62.3%
text-embedding-ada-002 (Published MTEB result; evaluation dimensions unspecified)OpenAI61%

Time and cost

Intelligence Index / Cost per task15 results

Base model resultArtificial Analysis

Weighted cost per evaluation task

USDLower is better
  1. GLM 5.3 Flash0.18
  2. Muse Spark 1.20.55
  3. Gemini 3.8 Flash0.74
  4. GPT-5.6 Terra0.81
  5. Muse Spark 1.30.96
  6. Qwen3.8 2.4T A95B1.1
  7. Qwen3.8 Max1.19
  8. GPT-5.6 Sol1.25
  9. Grok 4.61.25
  10. GLM-5.31.26
  11. Kimi K31.58
  12. GPT-6 Astra2.57
  13. Claude Opus 54.21
  14. Claude Fable 55.62
  15. Claude Fable 5.16.12
Intelligence Index / Cost per task / USD
Model and configurationScore
GLM 5.3 Flash (GLM-5.3-Flash)Z AI0.18
Muse Spark 1.2 (xhigh)Meta0.55
Gemini 3.8 Flash (high)Google0.74
GPT-5.6 Terra (max)OpenAI0.81
Muse Spark 1.3 (max)Meta0.96
Qwen3.8 2.4T A95BAlibaba1.1
Qwen3.8 MaxAlibaba1.19
GPT-5.6 Sol (max)OpenAI1.25
Grok 4.6 (high)SpaceXAI1.25
GLM-5.3 (max)Z AI1.26
Kimi K3 (max)Kimi1.58
GPT-6 Astra (max)OpenAI2.57
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic4.21
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic5.62
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic6.12
Output speed15 results

Base model resultArtificial Analysis

Output generation speed

tokens/sHigher is better
  1. Gemini 3.8 Flash280.76
  2. Muse Spark 1.2269.24
  3. Muse Spark 1.3233.15
  4. GPT-5.6 Terra113.93
  5. GLM-5.383.31
  6. GPT-5.6 Sol75.78
  7. Claude Fable 5.168.22
  8. GPT-6 Astra64.26
  9. Claude Fable 562.97
  10. Grok 4.659.89
  11. Claude Opus 555.95
  12. GLM 5.3 Flash50.63
  13. Kimi K341.61
  14. Qwen3.8 2.4T A95B40.23
  15. Qwen3.8 Max40.15
Output speed / tokens/s
Model and configurationScore
Gemini 3.8 Flash (high)Google280.76
Muse Spark 1.2 (xhigh)Meta269.24
Muse Spark 1.3 (max)Meta233.15
GPT-5.6 Terra (max)OpenAI113.93
GLM-5.3 (max)Z AI83.31
GPT-5.6 Sol (max)OpenAI75.78
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic68.22
GPT-6 Astra (max)OpenAI64.26
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic62.97
Grok 4.6 (high)SpaceXAI59.89
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic55.95
GLM 5.3 Flash (GLM-5.3-Flash)Z AI50.63
Kimi K3 (max)Kimi41.61
Qwen3.8 2.4T A95BAlibaba40.23
Qwen3.8 MaxAlibaba40.15
API input price15 results

Base model resultArtificial Analysis

Price per million input tokens

USD / 1MLower is better
  1. GLM 5.3 Flash0.15
  2. Gemini 3.8 Flash0.75
  3. Muse Spark 1.31.25
  4. Muse Spark 1.21.25
  5. GLM-5.31.4
  6. Grok 4.62
  7. Qwen3.8 Max2
  8. GPT-5.6 Terra2
  9. Qwen3.8 2.4T A95B2
  10. Kimi K33
  11. GPT-5.6 Sol4
  12. Claude Opus 55
  13. Claude Fable 5.110
  14. GPT-6 Astra10
  15. Claude Fable 510
API input price / USD / 1M
Model and configurationScore
GLM 5.3 Flash (GLM-5.3-Flash)Z AI0.15
Gemini 3.8 Flash (high)Google0.75
Muse Spark 1.3 (max)Meta1.25
Muse Spark 1.2 (xhigh)Meta1.25
GLM-5.3 (max)Z AI1.4
Grok 4.6 (high)SpaceXAI2
Qwen3.8 MaxAlibaba2
GPT-5.6 Terra (max)OpenAI2
Qwen3.8 2.4T A95BAlibaba2
Kimi K3 (max)Kimi3
GPT-5.6 Sol (max)OpenAI4
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic5
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic10
GPT-6 Astra (max)OpenAI10
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic10
API output price15 results

Base model resultArtificial Analysis

Price per million output tokens

USD / 1MLower is better
  1. GLM 5.3 Flash0.5
  2. Gemini 3.8 Flash3.75
  3. Muse Spark 1.34.25
  4. Muse Spark 1.24.25
  5. GLM-5.34.4
  6. Grok 4.66
  7. Qwen3.8 Max6
  8. Qwen3.8 2.4T A95B6
  9. GPT-5.6 Terra12
  10. Kimi K315
  11. GPT-5.6 Sol20
  12. Claude Opus 525
  13. Claude Fable 5.150
  14. GPT-6 Astra50
  15. Claude Fable 550
API output price / USD / 1M
Model and configurationScore
GLM 5.3 Flash (GLM-5.3-Flash)Z AI0.5
Gemini 3.8 Flash (high)Google3.75
Muse Spark 1.3 (max)Meta4.25
Muse Spark 1.2 (xhigh)Meta4.25
GLM-5.3 (max)Z AI4.4
Grok 4.6 (high)SpaceXAI6
Qwen3.8 MaxAlibaba6
Qwen3.8 2.4T A95BAlibaba6
GPT-5.6 Terra (max)OpenAI12
Kimi K3 (max)Kimi15
GPT-5.6 Sol (max)OpenAI20
Claude Opus 5 (Adaptive Reasoning, Max Effort)Anthropic25
Claude Fable 5.1 (Adaptive Reasoning, Max Effort, Default Fallback)Anthropic50
GPT-6 Astra (max)OpenAI50
Claude Fable 5 (Adaptive Reasoning, Max Effort, Opus 4.8 Fallback)Anthropic50

Sources and further reading

  1. Anthropic. Demystifying evals for AI agents
  2. OpenAI. GDPval
  3. OpenAI. BrowseComp
  4. Parallel. Deep Research price-performance
  5. LangChain. Improving Deep Agents with harness engineering
  6. LangChain. Deep Agents source code
  7. DeepResearch Bench II. Evaluation code
  8. ResearchRubrics. Evaluation code
  9. Agents’ Last Exam. Tasks and evaluation