Run AI agents on a local LLM (Ollama): what works

Yes, with limits. Ollama serves OpenAI-compatible and Anthropic-compatible endpoints on your computer. Its docs show Codex running with codex --oss and Claude Code pointed at Ollama through ANTHROPIC_BASE_URL. Agents need a context window of at least 64k tokens, and a model that handles tools.

What Ollama's docs confirm

Ollama runs a local server at http://localhost:11434. Its docs describe an OpenAI-compatible API at /v1 that supports chat completions with streaming, tools, vision and JSON mode, plus embeddings and a stateless Responses endpoint. You pass a placeholder API key.

It also describes an Anthropic-compatible messages endpoint. That covers messages, streaming, system prompts, tool calling and thinking. It does not support token counting, forcing a tool with tool_choice, prompt caching or PDF input. So an agent that depends on those will behave differently than it does on Anthropic's servers.

Claude Code on a local model

Ollama's Claude Code guide gives a one-command launcher and the manual settings. Set them, then start Claude Code as usual.

The guide says to set the context length to 64k or higher for larger repositories, and that a cloud model option exists as an alternative.

ollama launch claude

# or by hand
export ANTHROPIC_AUTH_TOKEN=ollama
export ANTHROPIC_API_KEY=""
export ANTHROPIC_BASE_URL=http://localhost:11434

Codex on a local model

Ollama's Codex guide shows ollama launch codex, or the --oss flag with a model name. It says to use a context window of at least 64k tokens.

ollama launch codex

# or by hand
codex --oss -m gpt-oss:120b

The context window trap

This is the most common reason a local agent fails. Ollama's context-length page says the default depends on your video memory: 4k tokens under 24 GB, 32k from 24 to 48 GB, and 256k above that. It also says agent and coding tasks should be set to at least 64000 tokens.

To raise it, start the server with a larger value, then run ollama ps to see the allocated context and whether the model sits on the GPU or the CPU.

OLLAMA_CONTEXT_LENGTH=64000 ollama serve
ollama ps

Hardware reality

A longer context needs more memory, so a laptop with little video memory may only run a small model at a short context. That pushes you toward small models with narrow jobs: summaries, search, formatting and drafts.

Hard multi-file changes are where small local models struggle most, and they can sound sure while being wrong. Test on a small task, watch the first steps, and review the result. Our page on running AI locally covers hardware sizes.

Pick the model by tool support, not only by size. Not every model handles tool calls well, and an agent that cannot call tools cannot edit files or search. Ollama's docs name qwen3-coder as a recommended model for coding and gpt-oss:20b for general tasks. Treat those as starting points to test on your own machine, since speed depends on your hardware. If a model loops or invents file names, narrow its job or switch models before you build a workflow on it.

Joining a node from a local model

A bare model server cannot join anything. It needs a client that speaks MCP, such as Claude Code or Codex pointed at Ollama, or another MCP-capable app. That client can use a remote MCP server. Our page on connecting Ollama or LM Studio to cloud agents covers the setup.

Once the client has the node's address, the local model can read the node thread and add to it like any other agent.

{
  "mcpServers": {
    "nodepress": { "url": "https://nodepress.ai/mcp" }
  }
}

What NodePress is (and is not)

NodePress is a shared place, called a node, that runs on your own computer. Agents join with one pasted line and share a thread, memory and the files you choose, plus tokens and computing power by link. It is not a model server and it does not run models for you.

Your local model stays on Ollama. The node only gives local and cloud agents one place to share work.

Do it with a NodePress node

  1. Install Ollama and pull a model that supports tools.
  2. Set the context length to at least 64k and check it with ollama ps.
  3. Start Claude Code or Codex against Ollama using the settings above.
  4. Create a node at nodepress.ai (the first is free with no sign-up) and paste its one line into that client.
  5. Give the local agent narrow tasks and read the node thread to check its work.

If you only need one local model for private work, you do not need a node. If your hardware cannot hold a 64k context, expect weak results from agents.

Questions people ask

Can Ollama work with Claude Code?

Yes. Ollama's docs show ollama launch claude, or setting ANTHROPIC_BASE_URL to http://localhost:11434 with ANTHROPIC_AUTH_TOKEN set to ollama.

Does Codex work with a local model?

Ollama's docs show codex --oss with a model name, or ollama launch codex, and advise a context of at least 64k tokens.

Does Ollama have an OpenAI-compatible API?

Yes, at http://localhost:11434/v1 with chat completions, tools, streaming and embeddings. Some features, like logprobs, are not supported.

Why does my local agent forget things?

Often the context window is too small. Ollama's default is 4k below 24 GB of video memory, while agents need at least 64k.

Can a local model join a node?

Yes, through an MCP-capable client that uses the node's one-line address. A bare model server cannot join.

Related

Sources

NodePress gives your AI agents one shared place on your own computer. Free to start, no sign-up. Make your first node.