llama-swap

What it is

llama-swap is an open-source, lightweight model proxy and routing engine designed to hot-swap local GGUF models on demand behind a single OpenAI-compatible API endpoint. Built primarily for resource-constrained home labs, edge deployments, and developer workstations running llama.cpp backends, it automatically manages loading, serving, and unloading model processes based on incoming API request parameters.

By managing backend process lifecycles and isolating model loading from client API logic, llama-swap provides an intelligent abstraction layer for hosting multiple LLMs on consumer hardware.

What problem it solves

Running multiple local LLMs concurrently requires significant VRAM or RAM, which quickly exhausts hardware capacity on consumer GPUs or single-board computers. Manually starting and stopping different model server instances creates operational friction, consumes admin effort, and breaks automated agent workflows that expect an always-available, unified API endpoint.

llama-swap solves this by transparently intercepting OpenAI-compatible API calls, inspecting the requested model parameter, gracefully unloading active models from GPU memory, and spinning up requested models dynamically—all without requiring manual server reconfiguration or breaking client connections.

Where it fits in the stack

Infrastructure / Model Routing & Serving. llama-swap sits between AI clients, agents, and IDE extensions and underlying llama.cpp server backends, acting as a dynamic, VRAM-conscious proxy and process orchestration layer.

graph TD
    Client[AI Client / FastMCP / Open WebUI] -->|HTTP POST /v1/chat/completions| Proxy[llama-swap Proxy Engine]

    subgraph llama-swap Router
        Proxy --> Config[YAML Config / Registry]
        Proxy --> Monitor[TTL Inactivity Monitor]
    end

    Proxy -->|Load Request Model / Unload Old| Backend[llama-server Manager]
    Backend -->|Spawn / Terminate| Server1[llama-server: Model A]
    Backend -->|Spawn / Terminate| Server2[llama-server: Model B]
    Server1 -->|Metal / CUDA| VRAM[GPU VRAM Shared Pool]
    Server2 -->|Metal / CUDA| VRAM

Typical use cases

• Multi-Model Home Lab Serving: Hosting specialized code-completion, reasoning, and conversational models on a single GPU without triggering VRAM out-of-memory errors.
• OpenAI API Proxying: Intercepting requests from tools like Open WebUI, LiteLLM, LibreChat, or Cursor to dynamically load target models on demand.
• Automated Memory Reclamation: Automatically unloading idle models after a configurable Time-To-Live (TTL) timeout to free system resources for other workloads.
• Multi-Agent Orchestration: Allowing multi-agent workflows (e.g. CrewAI, AutoGen, or FastMCP pipelines) to switch between different model sizes (7B, 14B, 32B) seamlessly.

Strengths

• VRAM & Hardware Optimization: Maximizes memory efficiency by ensuring only active models reside in VRAM.
• Drop-in OpenAI Compatibility: Works seamlessly with standard OpenAI client SDKs, LangChain, LlamaIndex, and agent frameworks.
• Zero-Downtime Configuration: Allows dynamic model registry updates without restarting the proxy engine.
• Resource Reclaim Automation: Automatically unloads idle models after custom inactivity timeouts to conserve electricity and memory.
• Transparent Request Buffering: Queues incoming requests during model swaps to prevent client-side connection drops.

Limitations

• Load Latency Penalty: Switching models introduces initial startup latency while loading multi-gigabyte GGUF weights into system/GPU memory.
• Single-User / Low-Concurrency Focus: Optimized for individual home labs, developer workstations, or small teams rather than multi-tenant high-throughput production clusters.
• Process Management Dependencies: Requires valid local executable paths for underlying llama-server or llama.cpp binaries.

When to use it

• When operating home lab hardware with limited VRAM (e.g., 8GB–24GB GPUs) that cannot hold multiple models simultaneously.
• When agent workflows require access to specialized models (e.g., Qwen-Coder vs Llama-3) on demand.
• When requiring automatic idle model unloading to conserve power and VRAM.
• When seeking a unified OpenAI API endpoint for a collection of local GGUF models.

When not to use it

• When running enterprise inference clusters where all models must remain warm with zero load latency.
• When using high-concurrency model servers like vLLM or TGI that manage continuous batching across shared multi-GPU setups.

Getting started

Installation & Execution

To set up llama-swap on a home lab server:

# Install llama-swap CLI / binary via Go
go install github.com/sammcj/llama-swap@latest

# Start llama-swap with a configuration file
llama-swap --config config.yaml

Configuration Example

Example config.yaml:

port: 8080
ttl: 300 # Unload model after 5 minutes of inactivity
models:
  llama-3:
    cmd: "llama-server -m /models/llama-3-8b.gguf --port 8081"
    port: 8081
  qwen-coder:
    cmd: "llama-server -m /models/qwen2.5-coder-7b.gguf --port 8082"
    port: 8082

CLI examples

# Check running llama-swap status and active model
llama-swap status

# Pre-warm a specific model endpoint via HTTP POST
curl -X POST http://localhost:8080/v1/models/llama-3/load

# Send a chat completion request that triggers automatic model swap
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen-coder",
    "messages": [{"role": "user", "content": "Write a Python script to sort a list."}]
  }'

API examples

1. Pydantic v2 Schema for llama-swap Configuration Validation

from typing import Dict, Optional
from pydantic import BaseModel, ConfigDict, Field

class ModelEndpointConfig(BaseModel):
    model_config = ConfigDict(extra="forbid")

    cmd: str = Field(..., description="Execution command for launching backend llama-server instance")
    port: int = Field(..., ge=1024, le=65535, description="Local port for the backend instance")
    args: Optional[list[str]] = Field(default_factory=list, description="Additional CLI arguments for llama-server")

class LlamaSwapConfig(BaseModel):
    model_config = ConfigDict(extra="forbid")

    port: int = Field(default=8080, ge=1024, le=65535, description="Proxy listening port")
    ttl_seconds: int = Field(default=300, ge=0, description="Idle timeout before unloading model")
    models: Dict[str, ModelEndpointConfig] = Field(..., description="Registry of available models")

if __name__ == "__main__":
    cfg = LlamaSwapConfig(
        port=8080,
        ttl_seconds=300,
        models={
            "qwen-coder": ModelEndpointConfig(
                cmd="llama-server -m /models/qwen2.5-coder.gguf --port 8081",
                port=8081,
                args=["-c", "8192", "-ngl", "99"]
            ),
            "llama-3": ModelEndpointConfig(
                cmd="llama-server -m /models/llama-3-8b.gguf --port 8082",
                port=8082,
                args=["-c", "4096", "-ngl", "99"]
            )
        }
    )
    print(f"llama-swap configured on port {cfg.port} with {len(cfg.models)} model(s).")
    print("Serialized Config:\n", cfg.model_dump_json(indent=2))

2. FastMCP 3.1 Task Protocol Integration

from typing import Dict, Any
from mcp.server.fastmcp import FastMCP
import urllib.request
import json

mcp = FastMCP("llama-swap-manager")

PROXY_URL = "http://localhost:8080"

@mcp.tool()
def swap_model(model_name: str) -> Dict[str, Any]:
    """Pre-loads or swaps target GGUF model via llama-swap proxy endpoint."""
    url = f"{PROXY_URL}/v1/models/{model_name}/load"
    req = urllib.request.Request(url, method="POST")
    try:
        with urllib.request.urlopen(req, timeout=30) as resp:
            data = json.loads(resp.read().decode("utf-8"))
            return {"status": "success", "requested_model": model_name, "details": data}
    except Exception as e:
        return {"status": "error", "message": str(e)}

@mcp.tool()
def get_proxy_status() -> Dict[str, Any]:
    """Queries current status and loaded model in llama-swap proxy."""
    url = f"{PROXY_URL}/status"
    try:
        with urllib.request.urlopen(url, timeout=10) as resp:
            data = json.loads(resp.read().decode("utf-8"))
            return {"status": "success", "info": data}
    except Exception as e:
        return {"status": "error", "message": str(e)}

if __name__ == "__main__":
    mcp.run()

Related tools / concepts

• llama.cpp — Core backend execution engine for GGUF models.
• Ollama — Alternative local model runner with built-in model management.
• Model Routing Guide — Architectural patterns for local model routing.
• LM Studio — Cross-platform desktop local inference application.

Sources / references

• llama-swap GitHub Repository

---

Contribution Metadata

• Last reviewed: 2027-01-07
• Confidence: high