Skip to content

Terminal-Bench (Terminus 2)

What it is

Terminal-Bench (including the Terminus 2 research baseline) is a benchmark for evaluating AI agents' ability to use a terminal. It focuses on tasks that require interacting with a real terminal environment, such as installing software, debugging system issues, managing files, and direct LLM-to-tmux shell interaction—a key June 2026 capability for autonomous DevOps agents.

What problem it solves

Measures whether AI agents can effectively operate in a terminal environment, a critical capability for autonomous system administration and DevOps tasks. It goes beyond code generation by testing the agent's ability to interpret command output, handle stateful sessions, and remediate system failures.

Where it fits in the stack

Benchmarking. Used to evaluate AI agents on terminal-based tasks within agentic orchestration layers. It is the primary benchmark for "Terminus 2" patterns where agents manage long-running tmux sessions.

Typical use cases

  • DevOps Agent Evaluation: Testing agents on terminal interaction tasks (installation, debugging, file management) before deployment to production.
  • Comparison of Agent Frameworks: Assessing how different frameworks (e.g., OpenHands, Aider) handle real-world system environments.
  • Autonomous SysAdmin Research: Assessing the readiness of AI agents for high-stakes autonomous system administration and security patching.
  • Multi-step Trajectory Analysis: Evaluating an agent's ability to maintain state across multiple shell commands.

Strengths

  • Practical Realism: Tests practical, real-world terminal skills rather than abstract coding problems or synthetic laboratory examples.
  • Stateful Interaction: Focuses on the "Intent-State" loop, requiring agents to observe and react to dynamic system changes.
  • DevOps Alignment: Covers a range of tasks directly relevant to modern SRE and DevOps workflows in June 2026.
  • Direct tmux Support: Terminus 2 specializes in direct LLM interaction with terminal multiplexers for persistent session management.

Limitations

  • Environment Complexity: Requires a real or containerized terminal environment for evaluation, adding significant setup overhead.
  • OS Specificity: Results may vary depending on the operating system and environment configuration (e.g., Ubuntu vs. Alpine).
  • Flakiness: Like web-benchmarks, terminal-bench can be subject to environmental flakiness if dependencies are not strictly pinned.

When to use it

  • When evaluating AI agents that need to operate autonomously in terminal environments for June 2026 DevOps tasks.
  • When assessing system administration or security patching capabilities of frontier models (Claude 4.8, GPT-5.5).
  • When researching persistent session management (tmux) for agents.

When not to use it

  • When evaluating pure code generation capabilities (use HumanEval or MBPP).
  • When you need a lightweight, fast-running benchmark for early-stage development (requires Docker/Harbor).
  • For evaluating high-level visual reasoning (use ColQwen or AssistantBench).

Getting started

Terminal-Bench (TB-2) is typically run using the Harbor framework to provide a consistent, containerized execution environment.

1. Installation

pip install terminal-bench harbor-framework
# Ensure Docker is installed and running

2. Configuration

Configure the Harbor sandbox for agentic evaluation: 

harbor init --benchmark terminal-bench

CLI examples

Running a specific DevOps task

Evaluate an agent's ability to set up an Nginx load balancer: 

tb run --task_id "nginx-lb-config" --model "anthropic/claude-4-8-sonnet"

Listing available benchmarks

tb list --category "sysadmin"

Managing tmux sessions via Terminus 2

Terminus 2 allows for direct shell interaction: 

terminus2 connect --session "devops-audit" --agent "my-devops-droid"

API examples

Orchestrating Terminal Evaluation

Using the Harbor framework to run sandboxed evaluations in June 2026.

from harbor import HarborSandbox, TerminalBenchTask
from my_agent import TerminalAgent

# Initialize sandboxed environment
with HarborSandbox(image="harbor/ubuntu-22.04-dev") as sandbox:
    # Define the terminal task
    task = TerminalBenchTask(
        id="debug-c-memory-leak",
        instruction="Find and fix the memory leak in the provided C application."
    )

    # Execute agent in the sandbox
    result = sandbox.execute_agent(
        agent=TerminalAgent(model="gpt-5-5"),
        task=task,
        timeout=600
    )

    print(f"Task Completed: {result.success}")
    print(f"Agent Trajectory: {result.trajectory_log}")

Direct tmux Interaction (Terminus 2 Pattern)

from terminus2 import TmuxSession

# Open a persistent session for the agent
with TmuxSession(name="agent-workspace") as session:
    output = session.send_command("ls -R /etc/nginx")
    # Agent reasons over output and sends next command...
    session.send_command("vim /etc/nginx/nginx.conf")

Sources / references