I Built an AI Agent That Works With No Internet. Here's How.

I wanted an AI agent that works with no internet and keeps everything private. No data leaving the machine. No API calls logged somewhere else. No cloud service deciding what stays and what gets stored.

So I built it. A local AI agent in Rust, running Gemma 4 from a GGUF file on disk. No network calls. No API keys. No cloud dependency. One binary, any machine.

The stack

Rust handles the inference engine, routing, security, and HTTP server. The model itself runs through llama-cpp-2 — safe Rust bindings to llama.cpp, which is the same engine Ollama uses under the hood. The model file is a GGUF — a single binary blob you drop into a models/ folder.

llama-cpp-2  = "0.1"          # llama.cpp bindings
axum         = { version = "0.7", features = ["multipart"] }  # HTTP server
tokio        = { version = "1", features = ["full"] }          # async runtime
aes-gcm      = "0.10"          # model encryption
zeroize      = { version = "1", features = ["derive"] }        # memory wipe
chrono       = { version = "0.4", features = ["serde"] }       # audit timestamps

No Python. No PyTorch. No virtual environments. cargo build --release and you're done.

Two models, one pool

The same Gemma 4 E2B model ships in two quantizations. Q4_K_M is smaller and faster — good for general questions, document Q&A, legal and financial summaries. Q8_0 is more precise — used for anything where a wrong answer matters more than speed: medical records, compliance audits, safety classification.

const MODEL_PATHS: &[(&str, &str)] = &[
    ("q4", "models/gemma4-e2b-q4.gguf"),
    ("q8", "models/gemma4-e2b-q8.gguf"),
];

pub const CTX_SIZE: usize = 131_072;

Both load at startup. If you only have one, the pool falls back automatically. Context window is 131,072 tokens — enough for most contracts, reports, and medical documents without chunking.

Nine agents

Every agent is a Rust enum variant. Each one has a system prompt baked into the binary at compile time using include_str!, a model tier, and a string ID for the API. Adding a new agent means adding a variant — the compiler tells you every match arm you missed.

Safety-critical agents use Q8 because a wrong answer in a medical or compliance context costs more than the extra inference time.

Two-stage routing

When a request comes in without a specified agent, the system routes it automatically. But calling an LLM to classify every request adds latency. So routing is two stages.

Stage one is a set of hard guards — deterministic keyword checks that resolve obvious cases in zero milliseconds. If the query contains "toxic" or "harmful", it goes to the safety agent. If it contains "classify" or "sentiment", it goes to the classifier. No model call needed.

pub fn guard_route(query: &str, has_document: bool, has_history: bool) -> Option<AgentId> {
    if has_history && !has_document {
        return Some(AgentId::General);
    }
    let q = query.to_lowercase();
    if q.contains("classify") || q.contains("sentiment") {
        return Some(AgentId::Classifier);
    }
    if q.contains("toxic") || q.contains("harmful") {
        return Some(AgentId::Safety);
    }
    None  // fall through to LLM router
}

If the guards don't match, stage two runs. The router summarizes the query into a short task description — stripping the actual content so the router only sees intent — and sends it to Gemma itself using the Q4 model. Gemma responds with a single agent ID. If the response is unrecognizable, it falls back to General.

Security I didn't skip

Running a model locally means the GGUF file sits on disk. If you're using this on a shared machine or shipping it to a client, you probably don't want the weights readable by anyone who can access the filesystem.

The binary can encrypt the model file at rest using AES-256-GCM with a key from an environment variable. At startup, if it finds a .enc file and no plaintext, it decrypts to a temp file, loads it, then the temp file gets cleaned up on exit.

// Encrypt once:
gemma4-agent --encrypt-model models/gemma4-e2b-q4.gguf
// GEMMA_MODEL_KEY env var required

// At startup — automatic, transparent:
resolve_model_path("models/gemma4-e2b-q4.gguf")
// → finds .enc, decrypts, loads, cleans up

After loading, the model weights get locked in RAM using mlockall() on Unix. This tells the OS not to swap those pages to disk. The weights stay in memory, not on the swap partition.

Output validation runs after every inference for medical, legal, compliance, and safety agents. The medical agent flags prescriptive language like "you should take" or "stop taking" and checks that the answer cites the source document. The compliance agent flags any output that says "fully compliant" or "100% compliant" without qualification. The safety agent blocks any output that starts explaining harmful content.

Every request gets written to an audit log as a JSON line: timestamp, agent, model tier, document size, query preview, answer hash, warnings, blocked flag, latency, token counts. The answer itself is hashed, not stored.

Document input

The agent takes a file path, reads it, and extracts text before inference. Four formats work: PDF, DOCX, CSV, and plain text. In server mode, file uploads come as multipart form data and get parsed from bytes in memory — nothing writes to disk.

# CLI mode
gemma4-agent \
  --agent legal \
  --doc contract.pdf \
  --query "What are the termination clauses?"

# Server mode
gemma4-agent --serve --port 8080

# Cluster mode — 3 workers + load balancer
gemma4-agent --serve --port 8090 --instances 3

What I learned

• llama-cpp-2 makes local inference in Rust straightforward — the hard work is already done in llama.cpp
• Two-stage routing is worth it: most requests resolve in stage one with zero LLM calls
• Q4 vs Q8 is a real tradeoff — for safety-critical agents, the accuracy difference matters more than the speed difference
• mlockall() is one line of code that prevents weights from leaking to swap — there's no reason not to use it
• Baking prompts into the binary with include_str! means one less config file to distribute and one less thing to misconfigure

The whole thing compiles to a single static binary. You can copy it to any Linux or macOS machine, add the GGUF file, and it runs. No runtime dependencies, no Python version conflicts, no package managers. That was the original goal. It works on a plane.

Open questions