GPQA (Graduate-Level Google-Proof Q&A)¶

What it is¶

GPQA is a challenging benchmark for evaluating high-level reasoning and knowledge in LLMs. It consists of 448 multiple-choice questions written by experts (PhD-level) in biology, physics, and chemistry. The questions are designed to be "Google-proof," meaning they are difficult even for non-expert humans to solve with access to the internet. As of June 2026, it remains a critical metric for frontier reasoning models like Claude 4.8 and GPT-5.5.

What problem it solves¶

Measures whether LLMs possess deep, expert-level scientific knowledge and reasoning that cannot be trivially looked up, providing a more rigorous assessment than general knowledge benchmarks like MMLU which are increasingly appearing in training sets (contamination).

Where it fits in the stack¶

Benchmarking. Used as a reference benchmark for evaluating advanced reasoning and scientific competence in state-of-the-art LLMs.

Typical use cases¶

Evaluating LLM performance on graduate-level scientific reasoning.
Comparing models on tasks that require genuine understanding rather than surface-level retrieval.
Assessing progress toward expert-level AI capabilities in STEM fields.
Validating the effectiveness of reasoning-heavy models like Claude 4.8 Opus for complex research.

Strengths¶

Expert-Verified: Questions are written and verified by PhD-level experts.
Search Resistant: Designed to be genuinely difficult even with internet access.
Broad Disciplines: Covers Biology, Physics, and Chemistry.
High Correlation: Strongly correlates with actual reasoning ability in scientific domains.

Limitations¶

Limited Scale: Small dataset (448 questions) may not cover all scientific sub-domains.
Domain Focus: Primarily "hard" sciences; does not cover humanities or social sciences.
Format: Multiple-choice format may not fully capture open-ended reasoning ability.
Expert Requirement: High barrier to entry for human verification (requires PhDs).

When to use it¶

When comparing frontier LLMs on their ability to handle difficult, expert-level scientific questions.
When you need a benchmark that is resistant to memorization and search-engine shortcuts.
To evaluate models designed specifically for deep research and reasoning.

When not to use it¶

When evaluating code generation or practical task completion (use HumanEval or SWE-bench).
When you need broad general-knowledge evaluation for a non-expert audience (use MMLU).
For testing basic conversational capabilities or "vibes" (use Chatbot Arena).

Getting started¶

GPQA is typically run using evaluation frameworks like the LM Evaluation Harness or proprietary evaluation pipelines for closed models.

Install the LM Evaluation Harness: pip install lm-eval
Prepare your model (Hugging Face or API-based).
Run the GPQA task against your model using the CLI.

CLI examples¶

1. Running GPQA via LM Evaluation Harness¶

Evaluate a local Hugging Face model on the primary GPQA dataset:

python -m lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-4-Maverick-70B \
    --tasks gpqa_diamond \
    --device cuda:0 \
    --batch_size 1

2. Evaluating an API-based model¶

Compare performance of a reasoning model via an API provider:

python -m lm_eval --model anthropic \
    --model_args model=claude-4-8-opus-20260528 \
    --tasks gpqa_main \
    --limit 50

3. Inspecting the dataset¶

Use huggingface-cli to download and inspect:

huggingface-cli download Idavidrein/gpqa --repo-type dataset

API examples¶

1. Python: Programmatic Evaluation¶

Using the lm_eval library to run benchmarks within a Python script:

import lm_eval

results = lm_eval.simple_evaluate(
    model="hf",
    model_args="pretrained=meta-llama/Llama-4-Maverick-70B",
    tasks=["gpqa_diamond"],
    num_fewshot=0
)

print(f"GPQA Diamond Accuracy: {results['results']['gpqa_diamond']['acc,none']:.2%}")

2. June 2026 Performance Metrics (Diamond)¶

Model	GPQA Diamond (Acc)	Release Date
Claude 4.8 Opus	74.5%	May 2026
GPT-5.5	71.2%	April 2026
Llama 4 Maverick	68.8%	June 2026
Claude 3.5 Sonnet	59.4%	June 2024
GPT-4o	53.6%	May 2024

3. Fetching Leaderboard Data via MCP¶

An agent might use an MCP tool to fetch the latest GPQA rankings:

{
  "tool": "get_benchmark_results",
  "arguments": {
    "benchmark": "gpqa",
    "category": "diamond"
  }
}

Sources / references¶

Contribution Metadata¶

Last reviewed: 2026-06-28
Confidence: high