Skip to content

MBPP (Mostly Basic Python Problems)

What it is

MBPP is a benchmark designed to evaluate the code generation performance of LLMs on basic Python tasks. It consists of approximately 1,000 crowd-sourced Python programming problems, designed to be solvable by entry-level programmers. Each problem includes a task description (prompt), a gold-standard code solution, and three automated test cases. It was introduced by Google Research in 2021 and remains a June 2026 baseline for agentic code generation.

What problem it solves

Provides a large-scale, standardized evaluation of LLM code generation on "mostly basic" problems. While benchmarks like HumanEval focus on algorithmic complexity, MBPP covers a broader range of fundamental programming concepts, standard library usage, and common data structure manipulations. It is a key metric for "Satisfaction-Based Validation" in June 2026 agentic software factories.

Where it fits in the stack

Benchmarking. Used as a primary code-generation benchmark for evaluating and comparing the Python coding capabilities of LLMs within agentic ingestion pipelines.

Typical use cases

  • Model Comparison: Measuring the Pass@1 and Pass@k metrics of new models (e.g., Claude 4.8, GPT-5.5) against industry baselines.
  • Fine-tuning Evaluation: Verifying that a model fine-tuned on code datasets (e.g., StarCoder 2026) has improved on basic programming tasks.
  • Contamination Testing: Using the "sanitized" version of the dataset to ensure results haven't been inflated by training data leakage—a critical requirement in June 2026.
  • Agent Skill Validation: Testing the core Python proficiency of autonomous agents before they are granted repository access.

Strengths

  • Large Dataset: With ~1,000 problems, it offers higher statistical confidence than smaller benchmarks like HumanEval.
  • Automated Verification: Each problem comes with executable test cases, ensuring objective, satisfaction-based scoring.
  • Sanitized Subset: A subset of the data has been hand-verified and "sanitized" to remove ambiguous or low-quality problems.
  • Realistic Basics: Focuses on tasks a junior developer or agent would perform, rather than just "LeetCode-style" puzzles.

Limitations

  • Basic Level: Does not evaluate architectural reasoning, multi-file projects, or advanced software engineering patterns (use SWE-bench for that).
  • Python Only: Limited to Python code generation.
  • Prompt Sensitivity: Results can vary based on the exact prompt format and "Thought" chain-of-thought (CoT) used by models like DeepSeek R1.
  • Saturation: High-end June 2026 models are reaching near 100% on MBPP, necessitating more difficult benchmarks like BigCodeBench.

When to use it

  • When evaluating the fundamental Python coding ability of a model or agent.
  • When you need a statistically robust code benchmark that is larger than HumanEval.
  • When assessing a model's familiarity with the Python standard library in June 2026.

When not to use it

  • When evaluating complex, real-world software engineering or repository-wide changes (use SWE-bench or BigCodeBench).
  • When testing non-Python languages (use MultiPL-E or similar).
  • When evaluating high-level agentic planning that isn't captured by "basic" problems.

Getting started

MBPP is typically run through evaluation frameworks like the LM Evaluation Harness or EvalPlus. In June 2026, it is often integrated into agentic CI/CD pipelines.

1. Installation

# Install via LM Evaluation Harness
pip install "lm_eval[hf,vllm]"

2. Basic Run

lm_eval --model vllm \
    --model_args pretrained=meta-llama/Llama-4-8b \
    --tasks mbpp \
    --batch_size auto

CLI examples

Evaluating a Sanitized Subset

lm_eval --model hf \
    --model_args pretrained=EleutherAI/pythia-160m \
    --tasks mbpp_sanitized \
    --device cuda:0

Running with LiteLLM Proxy

lm_eval --model openai-completions \
    --model_args model=gpt-5-5,base_url=http://localhost:4000 \
    --tasks mbpp \
    --limit 100

API examples

Programmatic Evaluation (Python)

Automate MBPP scoring within a June 2026 agentic workbench.

import lm_eval
from lm_eval.models.huggingface import HFLM

# Initialize model (e.g., for local verification)
model = HFLM(pretrained="deepseek-ai/deepseek-coder-7b-v1.5")

# Run evaluation on MBPP
results = lm_eval.simple_evaluate(
    model=model,
    tasks=["mbpp_sanitized"],
    num_fewshot=3,
    batch_size=16
)

# Extract Pass@1 score
pass_at_1 = results['results']['mbpp_sanitized']['pass@1']
print(f"DeepSeek MBPP Pass@1: {pass_at_1:.2%}")

Using EvalPlus for "Hardened" MBPP

EvalPlus adds thousands of extra test cases to MBPP to detect "fluke" passes.

from evalplus.data import get_mbpp
from evalplus.evaluate import evaluate

# Get hardened MBPP tasks
tasks = get_mbpp()

# Evaluate generated samples (e.g., from an agent)
results = evaluate(
    dataset="mbpp",
    samples="my_agent_samples.jsonl",
    test_setup="evalplus"
)
print(f"EvalPlus Hardened MBPP Score: {results['pass@1']}")

Sources / references