Skip to content

README.md

Latest commit

 

History

History
241 lines (176 loc) · 8.65 KB

README.md

File metadata and controls

241 lines (176 loc) · 8.65 KB
Raw

cat-agent-stack

Local HashiCorp stack for cat-agent deploy: Consul + Vault (-dev) + Nomad (server+client) + LiteLLM gateway (+ optional Zot).

Agent packaging / CLI live in the sibling cat-agent library (pip install 'cat-agent[platform]'). This repo is only the compose fixture and sample operator config.

Prerequisites

  • Docker Desktop (macOS) or Docker Engine (Linux)
  • pip install 'cat-agent[platform]' (provides cat-agent stack / deploy)
  • A real LLM backend for agent calls (default: host Ollama on Mac)

Quick start

cd /path/to/cat-agent-stack
cp .env.example .env
# edit .env — VAULT_TOKEN=root + Ollama/OpenAI

export CAT_AGENT_STACK_DIR=$PWD
export CAT_AGENT_CONFIG=$PWD/cat-agent.config.toml

cat-agent stack bootstrap     # compose up --build -d + Vault seed + demo team key
cat-agent doctor              # must print docker_network: cat-agent-stack_hashicorp

# deploy an agent from the cat-agent checkout:
cat-agent deploy --dir /path/to/cat-agent/examples/serve_fastapi
curl -sS http://demo-calculator.localhost:8088/readyz

Optional Zot registry profile:

cat-agent stack bootstrap --profile registry --registry

Tear down

cat-agent rm demo/calculator --yes   # optional
cat-agent stack down

Do not run bare docker compose — HOST_NOMAD_DATA / HOST_ZOT_DATA must be absolute paths; cat-agent stack sets them.

Keys: two layers (this is the confusing part)

There are two different kinds of secrets. Mixing them up feels like “aptallık”; they are not the same thing.

  laptop .env  ──seed──►  Vault platform LLM secret
                              │
                              ▼
                         LiteLLM gateway  ◄── uses Ollama + OpenAI provider keys
                              ▲
                              │  (virtual key as OPENAI_API_KEY)
  deployed agent job ─────────┘
         ▲
         └── reads ONE team virtual key from Vault
             secret/.../llm/teams/{team}

Layer A — provider keys (platform, shared)

Written by seed into secret/data/platform/llm.

Field Role
OLLAMA_API_BASE / OLLAMA_API_KEY Upstream Ollama (host or cloud)
OPENAI_API_BASE / OPENAI_API_KEY Upstream OpenAI (optional; can coexist with Ollama)
LITELLM_MASTER_KEY Admin key for LiteLLM (mint/delete virtual keys)

Yes: Ollama and OpenAI keys can both live here at once. LiteLLM picks which upstream to call from the model id in agent.yaml (model.alias), not from “which person deployed”.

These are org/stack credentials — not Ali’s vs Ayşe’s personal OpenAI keys.

Layer B — team virtual key (one per team)

Minted by LiteLLM (key/generate), stored only as api_key at secret/data/platform/llm/teams/{team} (default team: demo).

  • Every agent job for that team injects this same key as OPENAI_API_KEY.
  • The job’s OPENAI_BASE_URL points at the gateway, not at Ollama/OpenAI.
  • Purpose: gateway auth + TPM/RPM budget for that team, not identity of a person.

So today: 1 team → 1 virtual key → all that team’s deployed agents share it.

What we do not do today

Expectation Reality
Each person on the team has their own OpenAI/Ollama key in Vault Not supported for deploy. Provider keys are platform-shared (Layer A).
Each person has their own LiteLLM virtual key Not the default. Seed mints per team, not per user.
Laptop .env is what the Nomad job uses No. Jobs only see Vault (Layer B for the call key; LiteLLM uses Layer A).

If you need per-person billing / keys, that is a different product shape, e.g.:

  • mint LiteLLM keys per user (--team is a stand-in; you’d use teams/{user} or a users path), or
  • run cat-agent serve locally with that person’s own env keys (no Nomad), or
  • put provider keys behind a corporate gateway account and keep Layer B as team quotas only.

Local demo intent: one shared Ollama/OpenAI for the stack + one demo virtual key.

What is cat-agent stack seed?

Containers alone are not enough. Vault starts empty (-dev); seed fills Layer A

  • Layer B from this repo’s .env. It does not start/stop Docker.
Step What it does
1. Layer A Writes provider + master key into secret/data/platform/llm
2. Layer B Mints LiteLLM virtual key for --team (default demo) → …/teams/{team} + Vault read policy
3. Optional --registry → Zot push/pull secrets

Re-run seed after changing .env, recreating Vault, or adding another --team.

Commands (related)

Command Meaning
cat-agent stack up Start compose only
cat-agent stack seed Fill Vault + mint team key from .env (stack already up)
cat-agent stack bootstrap First-time: up --build -d then seed
cat-agent stack down Stop compose 
cat-agent stack seed                 # refresh Layer A + demo team key
cat-agent stack seed --team billing  # also mint a second team’s virtual key

Secrets are never printed; seed only logs bases / key lengths.

Config

File Role
cat-agent.config.toml Sample for CAT_AGENT_CONFIG (Nomad, gateway, Mac docker_network)
.env / .env.example Provider keys + VAULT_TOKEN=root for seed
docker-compose.yml Project name pinned: cat-agent-stack → network cat-agent-stack_hashicorp

macOS / Docker Desktop

Bridge CNI fails with unknown FS magic on "/var/run/docker/netns/…". This stack sets:

docker_network = "cat-agent-stack_hashicorp"
consul_dns = "10.32.0.2"

cat-agent doctor must show those lines before deploy.

Config loading (cat-agent)

Platform config lives here (not in the cat-agent library repo):

  1. CAT_AGENT_CONFIG
  2. $CAT_AGENT_STACK_DIR/cat-agent.config.toml
  3. cwd only if this directory is the stack root (docker-compose.yml present)
  4. sibling ../cat-agent-stack/cat-agent.config.toml next to a cat-agent checkout
  5. ~/.cat-agent/config.toml

Shared access (LAN / corp) — not *.localhost

*.localhost resolves to 127.0.0.1 on the client machine. Other PCs and phones on the same network cannot use http://demo-calculator.localhost:8088.

Traefik + Consul Catalog already front agents. For a corporate deployment, use your internal DNS (Active Directory / Bind / CoreDNS / cloud private zone) — do not rely on public wildcard DNS helpers.

# Must match what clients resolve via company DNS
ingress_host_template = "{team}-{name}.agents.example.internal"
public_url_template   = "https://{team}-{name}.agents.example.internal"

Then:

  1. Create a DNS record or wildcard zone so *.agents.example.internal → the Traefik / load-balancer VIP that publishes the agents (HTTP/HTTPS).
  2. Redeploy agents so Nomad job tags pick up the new Host(…) rule (cat-agent deploy again).
  3. Clients open the URL from public_url_template (any machine on the corp network that can resolve that name).

Placeholders: {team}, {name}, {agent} (agent == name).
Env overrides: CAT_AGENT_INGRESS_HOST_TEMPLATE, CAT_AGENT_PUBLIC_URL_TEMPLATE.

Local laptop default stays {team}-{name}.localhost — no DNS required on that Mac.

Day-2

cat-agent ls
cat-agent status demo/calculator
cat-agent logs demo/calculator
cat-agent rm demo/calculator --yes

Traefik (host :8088): hostname from ingress_host_template
(local default http://{team}-{name}.localhost:8088)
Dashboard: http://127.0.0.1:8089 · Nomad: http://127.0.0.1:4646

Contract with cat-agent

Knob Local value
llm_gateway http://llm-gateway.service.consul:4000/v1
vault_addr http://127.0.0.1:8200
VAULT_TOKEN root (-dev only)
Team key secret/data/platform/llm/teams/{team}
docker_network cat-agent-stack_hashicorp
consul_dns 10.32.0.2
ingress_host_template {team}-{name}.localhost
public_url_template http://{team}-{name}.localhost:8088
base_image cat-agent-runtime:latest (publish/pull in a later phase)

Layout

docker-compose.yml          # pinned name: cat-agent-stack
cat-agent.config.toml       # sample CAT_AGENT_CONFIG
.env.example
config/                     # consul, nomad, vault, litellm, zot
jobs/                       # gate Nomad jobs
Dockerfile.*                # consul / nomad / vault images
docker-entrypoint-*.sh      # container entrypoints (not operator scripts)

Linux notes

Omit or retarget Mac-only knobs (docker_network, consul_dns=10.32.0.2, host.docker.internal). See comments in cat-agent.config.toml and compose.