Skip to content

README.md

Latest commit

 

History

History
66 lines (43 loc) · 6.73 KB

README.md

File metadata and controls

66 lines (43 loc) · 6.73 KB
Raw

Local LLM

Runs models on this machine through one shared llama.cpp server, so nothing leaves the host and no provider usage window is spent.

Prerequisites

  • llama.cpp. brew install llama.cpp puts llama-server on the PATH. The module needs the router mode of the server (--models-preset, --models-max), which a current release has; it was built and measured on build 9960. Another binary can be named with LLAMA_SERVER_BIN.
  • Disk and memory. The three models below take about 49 GB on disk and the 70B needs about 40 GiB of GPU memory when loaded (figures measured on an M5 Max with 64 GB, where Metal allows 51.8 GiB). On a smaller machine, remove the 70B from LOCAL_MODELS (see Adding or removing a model) and keep the 8B and the embedding model, which together fit in about 14 GiB.
  • The models. Nothing is downloaded at install time. The first call that names a model makes the server download it from Hugging Face into ~/.local/share/cookbook/local-llm/models/, which can take minutes (the 70B is 37.9 GB); the tool reports progress while it waits. To fetch a model ahead of time, call llm_generate (or llm_rank for the embedding model) once with a one-word prompt and let it finish; llm_status shows the model as loaded or sleeping afterwards.

The models come from these Hugging Face repositories, each an ordinary GGUF repository that needs no account or token:

Model Repository (file chosen by the quantization after the colon)
llama-3.3-70b bartowski/Llama-3.3-70B-Instruct-GGUF :IQ4_XS
llama-3.1-8b bartowski/Meta-Llama-3.1-8B-Instruct-GGUF :Q8_0
qwen3-embedding-0.6b Qwen/Qwen3-Embedding-0.6B-GGUF :Q8_0

Tools

  • llm_status - each model's state (loaded, sleeping, loading, not downloaded), purpose, download size and context window, without loading anything.
  • llm_generate - answer one prompt; given jsonSchema, the answer is constrained to it, so it always parses.
  • llm_rank - order texts by meaning against a query with the embedding model.

How the server runs

The server runs in router mode on 127.0.0.1:8791: one process serves every model in src/modules/local-llm/config.ts, loads a model on the first request that names it (downloading it from Hugging Face the first time), and unloads a model after ten idle minutes. One chat model is loaded at a time, next to the embedding model: the 70B and the 8B together need more GPU memory than Metal allows (measured below), and a model that runs out is left broken, so loading a chat model first unloads the other, and a request still running on that one fails. A request whose GPU work fails anyway unloads its model, so the next call loads it fresh.

The first call that finds no server starts one, detached, so it outlives the caller and every session shares the models in memory; two callers starting at once both spawn, the port lets one bind, and both find it. Its files live in ~/.local/share/cookbook/local-llm/ (the model list it reads, server.log, and the downloaded models).

Model Download GPU memory loaded For
llama-3.3-70b 37.9 GB 40.5 GiB the most capable, and the slowest
llama-3.1-8b 8.5 GB 12.1 GiB fast classification and extraction
qwen3-embedding-0.6b 0.6 GB 1.8 GiB embeddings, for llm_rank and similarity

GPU memory is llama.cpp's own load-time breakdown (weights, the KV cache for the model's whole context, compute buffers) on an M5 Max with 64 GB, where Metal allows 51.8 GiB.

The server reads a JSON schema from OpenAI's response_format.json_schema.schema; its README's { type, schema } example is silently read as no schema. A first load of a large model takes minutes and reports progress; while it downloads, loading two other models would make the server unload it and cancel the download, so let a large first download finish alone.

Authentication

None. The models above download without a Hugging Face token (HF_TOKEN is only needed for a gated or private repository you add yourself), and the server binary comes from brew install llama.cpp.

Configuration

Variable Default Purpose
LOCAL_LLM_PORT 8791 The server's port
LLAMA_SERVER_BIN llama-server The llama.cpp server binary
XDG_DATA_HOME ~/.local/share Base of cookbook/local-llm/

Adding or removing a model

The models are the entries of LOCAL_MODELS in src/modules/local-llm/config.ts; the tools' model argument and llm_status follow that list. To add one:

  1. Find a GGUF repository on Hugging Face and pick a quantization. Repositories list one file per quantization (Q8_0 is near-lossless, Q4_K_M and IQ4_XS are smaller and faster); a repository that is gated or private needs HF_TOKEN in the environment of the process that starts the server.
  2. Add an entry with kind (chat or embedding), hfRepo as <user>/<repository>:<quantization>, sizeGb (the download), contextTokens, parallel (concurrent requests the model serves), memoryMib and a description. memoryMib is only reported by llm_status; take it from llama.cpp's load-time memory breakdown in the server log (~/.local/share/cookbook/local-llm/server.log) after a first load.
  3. Rebuild (pnpm run build). A running server rereads the list once per module process, so the next call picks the entry up without restarting the server, and models.ini in the data directory is rewritten from the config.

To drop a model, delete its entry. Its downloaded files stay in models/ until you delete the models--<user>--<repository> folder there.

Where it runs

  • Claude Code: not loaded by the bridge (isInClaudeCode: false in src/modules/registry.ts).
  • OpenCode: no agent allows llm_*. Every agent in opencode.json starts from deny-all and none lists these tools, and OpenCode's built-in agents are disabled, so in OpenCode the tools stay unreachable until an agent gets an allow rule for them.