Benchmarks

Tier 4 (hardest) FrontierMath problems - research-level difficulty

General science and observation benchmark

Balanced reasoning and logic games evaluation

Research-level mathematics problems at the frontier of current AI capabilities

Terminal/command-line interaction tasks for agent evaluation

Simple reasoning tasks designed to test fundamental capabilities

Multi-document synthesis and research tasks testing deep research capabilities

Cybersecurity CTF challenges testing security analysis and exploitation

Multi-step workplace automation tasks testing autonomous agent capabilities

Operating system interaction tasks testing computer use and automation

Abstract Reasoning Corpus - novel visual reasoning tasks testing general intelligence

Visual perception and comprehension tasks

Unusual machine learning tasks testing adaptability

Verified real-world GitHub issues requiring code understanding and modification

CAD/technical code evaluation tasks

Code editing tasks across multiple programming languages using Aider framework

Competition-level math problems from OTIS Mock AIME evaluating olympiad-level math

Graduate-level science questions in physics, chemistry, and biology requiring expert knowledge

Fiction comprehension and reasoning tasks

Adversarial NLI - challenging natural language inference

Level 5 (hardest) problems from the MATH dataset requiring advanced mathematical reasoning

Geographic and spatial reasoning tasks

Big-Bench Hard - 23 challenging tasks requiring multi-step reasoning

Science question answering across multiple domains

Massive Multitask Language Understanding across 57 subjects

Large-scale Winograd schema challenge - anchor benchmark for ECI calculation

Writing quality and style evaluation

Video understanding and multimodal evaluation tasks

Grade school math word problems requiring multi-step arithmetic reasoning

AI2 Reasoning Challenge - science questions requiring reasoning

Open-book question answering requiring common knowledge

Commonsense NLI about grounded situations

Physical intuition question answering

Trivia questions requiring broad knowledge

Language modeling and broad context understanding

Berkeley Function Calling Leaderboard - evaluates LLMs on tool/function calling accuracy across simple, multiple, parallel, and multi-turn scenarios

General AI Assistant benchmark - 466 tasks across 3 difficulty levels testing reasoning, web browsing, tool use, and multi-modality

GitHub issue resolution benchmark - tests coding agents on resolving real-world software issues with 500 human-verified instances