Skip to content

DREAM: Deep Research Evaluation with Agentic Metrics

What it is

DREAM (Deep Research Evaluation with Agentic Metrics) is an agentic evaluation framework for deep research agents. It uses tool-calling agents to independently verify the factual correctness and temporal validity of research reports. As of June 2026, it is the primary method for evaluating the "research depth" of models like Claude 4.8 Opus and GPT-5.5.

What problem it solves

It addresses the "Mirage of Synthesis"—a defect in static LLM evaluation where fluent writing and plausible citations hide factual errors or reasoning flaws. Static judges cannot verify claims against real-world evidence; DREAM solves this by making the evaluator as capable (agentic) as the agent it is testing, utilizing MCP 3.0 for tool discovery.

Where it fits in the stack

Eval / Benchmarking: It is a framework for benchmarking and evaluating advanced LLM agentic performance, particularly for models when used in complex research loops. It bridges the gap between static benchmarks like GPQA and real-world utility.

Typical use cases

  • Benchmarking Research Agents: Comparing how well different models or agent architectures (like OpenHands or custom research loops) generate accurate analyst-grade reports.
  • Reasoning Probes: Systematically identifying reasoning defects in long-form generation.
  • Fact-checking automation: Scaling the verification of AI-generated content against live web data using tools like Tavily.
  • Temporal Verification: Checking if a report's data is still valid or has been superseded by newer events.

Strengths

  • Parity-based Evaluation: Uses agents to evaluate agents, ensuring the evaluator has the tools necessary to verify modern information.
  • Sensitivity to Decay: Significantly more sensitive to factual and temporal decay than static benchmarks like MMLU.
  • Scalable and Reference-Free: Does not require a pre-defined ground truth for every query, allowing for flexible evaluation of open-ended research.
  • Tool-Agnostic: Can be integrated with any tool-calling environment supporting the Model Context Protocol.

Limitations

  • Operational Complexity: Requires a tool-calling environment for the evaluation agent, making it more complex to run than static Q&A.
  • Cost: Agentic evaluation involves multiple LLM turns, increasing the cost of benchmarking.
  • Latency: Evaluation can take several minutes per report, depending on the depth of the research required.

When to use it

  • When evaluating agents that perform active research or use external tools.
  • When static benchmarks are suspected of suffering from data contamination or lack of temporal awareness.
  • To validate the factual integrity of long-form reports generated by Claude Code or similar tools.

When not to use it

  • For evaluating simple base models on static knowledge where HumanEval is sufficient.
  • When a fast, low-cost evaluation signal is needed for iterative model tuning.
  • For non-research tasks like simple creative writing or translation.

Getting started

DREAM requires an environment where evaluation agents can execute search and browsing tools.

1. Installation

pip install dream-eval-framework

2. Environment Setup

Configure your LLM provider and search API keys. DREAM supports Claude 4.8 out of the box: 

export ANTHROPIC_API_KEY="your-key"
export TAVILY_API_KEY="your-key"

3. Basic Verification

Run a verification check on a local report file: 

dream-eval verify --report ./my_research_report.md

CLI examples

1. Running a Benchmark

Run the DREAM benchmark suite against a research agent endpoint: 

dream-eval benchmark --agent-url "http://localhost:8080/chat" --tasks research_tasks_v1.json

2. Temporal Sensitivity Check

Check if a report is outdated by forcing the agent to prioritize recent news sources: 

dream-eval verify --report report.md --temporal-weight 0.8

3. Exporting Results

Export the verification results to a structured JSON format for further analysis: 

dream-eval verify --report report.md --output results.json

API examples

Python Verification Loop

A conceptual implementation of the DREAM verification step using the Python SDK.

from dream_eval import DreamEvaluator

# Initialize the agentic evaluator with search tools via MCP
evaluator = DreamEvaluator(
    model="claude-4-8-opus-20260528",
    mcp_servers=["https://search.api.tavily.com/mcp"]
)

report_content = "..." # The output from the research agent

# DREAM starts its independent verification
verification_results = evaluator.verify_claims(report_content)

for claim in verification_results.claims:
    print(f"Claim: {claim.text}")
    print(f"Status: {claim.verification_status}") # Verified | Refuted | Unverifiable

Sources / references

Contribution Metadata

  • Last reviewed: 2026-06-30
  • Confidence: high