Skip to content

Commit 6eafdd8

Browse files
committed
config: custom ports for each engine
An engine can now be registered on a non-default port three ways: - [[engine]] accepts a port (and host) shorthand instead of a full url, so moving llama.cpp off :8080 is a one-line change. - [discovery] ports adds non-default ports to the probe sweep on localhost and every target; a new port is tried with every adapter, openai last. - [llama_swap] / [strata] move the built-in threads (and the NInfer backend that follows llama-swap's /running proxy) off their stock ports. Ports configured for the built-in threads are skipped by discovery, so they are never mistaken for a separate engine. Docs and tests updated.
1 parent 9d8e9a6 commit 6eafdd8

9 files changed

Lines changed: 317 additions & 21 deletions

File tree

‎README.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,8 @@ Every source is optional. Each poll degrades to "no data".
9797
| Strata `:8080` (`/v1/models`, `/metrics`, `/slots`) | second engine plus context slots |
9898
| any engine in `speculum.toml` | the same fields through the pluggable adapters |
9999

100+
Ports in this table are the defaults; each can be moved. See [Custom ports](#custom-ports).
101+
100102
## Configuration
101103

102104
Config is optional. With no file, Speculum probes the default ports on localhost and shows what it finds. Copy `speculum.example.toml` to `speculum.toml` in the repo root or in `~/.config/speculum/` to change anything.
@@ -119,6 +121,41 @@ The file sets the bind address and port, discovery targets, history retention, a
119121

120122
Adapters: `ollama`, `llamacpp`, `vllm`, `sglang`, `freetoken`, `koboldcpp`, `tabbyapi`, `localai`, `lmstudio`, `unsloth`, `openai`, `custom`. [`docs/ENGINE-COVERAGE.md`](docs/ENGINE-COVERAGE.md) explains how each adapter recognises a server and which engines stay unsupported.
121123

124+
### Custom ports
125+
126+
An engine does not have to sit on its stock port. There are three ways to point Speculum at another one; pick whichever fits.
127+
128+
**1. Give an engine a `port` (or `host`).** When you configure an engine, `port` and `host` stand in for a full `url`. This is the quickest way to move llama.cpp off 8080:
129+
130+
```toml
131+
[[engine]]
132+
name = "llama.cpp"
133+
type = "llamacpp"
134+
port = 8081 # host defaults to 127.0.0.1
135+
```
136+
137+
`host = "10.0.0.41"` watches another box, and `port` alone leaves the default host. A `url` still works and can be combined with `port` to override just the port.
138+
139+
**2. Add extra discovery ports.** Discovery normally probes the ports in the table under [What it watches](#what-it-watches). `[discovery] ports` adds ports to that sweep on localhost and every `targets` host, so an engine on a non-default port is found and registered like any other:
140+
141+
```toml
142+
[discovery]
143+
enabled = true
144+
ports = [8081, 9091] # probed with every adapter, generic openai last
145+
```
146+
147+
**3. Move the built-in threads.** `llama-swap`, `Strata` and the NInfer backend that follows llama-swap are watched by dedicated threads, not by discovery. Point them elsewhere with `[llama_swap]` and `[strata]`, using `url`, or `host` / `port`:
148+
149+
```toml
150+
[llama_swap]
151+
port = 9091 # default http://127.0.0.1:9090
152+
153+
[strata]
154+
url = "http://127.0.0.1:8090" # default http://127.0.0.1:8080
155+
```
156+
157+
NInfer has no port of its own: it rides the proxy llama-swap reports in `/running`, so fixing `[llama_swap]` is enough. Ports configured here are skipped by discovery, so they are never mistaken for a separate engine.
158+
122159
## Demo mode
123160

124161
`?demo` runs a seeded simulation with no backend. Serve the repo root any way you like and open `http://localhost:8792/?demo`:

‎collector/config.py‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,13 @@
1818

1919
DEFAULTS = {
2020
"server": {"host": "127.0.0.1", "port": 8792},
21-
"discovery": {"enabled": True},
21+
# `ports` adds non-default localhost/target ports to the discovery probe
22+
# list, for an engine running somewhere other than its stock port.
23+
"discovery": {"enabled": True, "ports": []},
24+
# Built-in threads (not the adapter scheduler). Override `url`, or give a
25+
# `host` / `port`, when llama-swap or Strata runs on a non-default port.
26+
"llama_swap": {"url": "http://127.0.0.1:9090"},
27+
"strata": {"url": "http://127.0.0.1:8080"},
2228
"history": {"retention_days": 30}, # 30, 60, 90 or 0 to turn off
2329
# Threshold alerts, each held for hold_s before it fires. vram_pct is high on
2430
# purpose: engines fill the card by design; the useful VRAM alert is a foreign process holding it.
@@ -53,7 +59,8 @@ def _clean_engine(e):
5359
or unknown tables (a [[engine]] with no type is an error, not a
5460
discovery candidate)."""
5561
out = {}
56-
for k in ("name", "type", "url", "parent", "api_key_env", "optional",
62+
for k in ("name", "type", "url", "host", "port", "parent",
63+
"api_key_env", "optional",
5764
"health", "models", "metrics", "slots", "map"):
5865
if k in e:
5966
out[k] = e[k]
@@ -76,6 +83,8 @@ def load(path=None):
7683
cfg = {
7784
"server": dict(DEFAULTS["server"]),
7885
"discovery": dict(DEFAULTS["discovery"]),
86+
"llama_swap": dict(DEFAULTS["llama_swap"]),
87+
"strata": dict(DEFAULTS["strata"]),
7988
"history": dict(DEFAULTS["history"]),
8089
"alerts": dict(DEFAULTS["alerts"]),
8190
"engine": [],
@@ -86,6 +95,8 @@ def load(path=None):
8695
raw = tomllib.loads(used.read_text(encoding="utf-8"))
8796
_merge(cfg["server"], raw.get("server"))
8897
_merge(cfg["discovery"], raw.get("discovery"))
98+
_merge(cfg["llama_swap"], raw.get("llama_swap"))
99+
_merge(cfg["strata"], raw.get("strata"))
89100
_merge(cfg["history"], raw.get("history"))
90101
_merge(cfg["alerts"], raw.get("alerts"))
91102
for e in raw.get("engine") or []:

‎collector/engines.py‎

Lines changed: 64 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -51,20 +51,45 @@
5151
]
5252

5353

54-
def discovery_targets(extra_hosts=()):
54+
def discovery_targets(extra_hosts=(), extra_ports=()):
5555
"""(host, port, adapter types) triples to probe: localhost on every port,
5656
then each extra host on the same port list. Localhost comes first so a
57-
local engine always wins its own port over a remote one."""
57+
local engine always wins its own port over a remote one.
58+
59+
`extra_ports` are non-default ports ([discovery] ports in speculum.toml).
60+
A port the table already knows keeps its ordered types; a new one is
61+
probed with every specific adapter, generic `openai` last."""
62+
ports = list(DISCOVERY)
63+
known = {p for p, _types in DISCOVERY}
64+
for p in extra_ports or ():
65+
try:
66+
p = int(p)
67+
except (TypeError, ValueError):
68+
continue
69+
if p and p not in known:
70+
known.add(p)
71+
ports.append((p, _all_discovery_types()))
5872
hosts = []
5973
for h in extra_hosts or ():
6074
h = str(h).strip()
6175
if h and h not in hosts and h not in LOCAL_HOSTS:
6276
hosts.append(h)
63-
out = [(LOCAL_HOST, port, types) for port, types in DISCOVERY]
64-
out.extend((h, port, types) for h in hosts for port, types in DISCOVERY)
77+
out = [(LOCAL_HOST, port, types) for port, types in ports]
78+
out.extend((h, port, types) for h in hosts for port, types in ports)
6579
return out
6680

6781

82+
def _all_discovery_types():
83+
"""Every discoverable adapter, each once, generic `openai` last. Used for
84+
ports [discovery] ports adds that the built-in table does not know."""
85+
out = []
86+
for _port, types in DISCOVERY:
87+
for t in types:
88+
if t not in out:
89+
out.append(t)
90+
return tuple([t for t in out if t != "openai"] + ["openai"])
91+
92+
6893
def parse_prom(text):
6994
"""Tiny Prometheus text parser: sum the samples of each metric name,
7095
ignore labels. NaN / non-numeric values are skipped."""
@@ -984,16 +1009,42 @@ def poll(self):
9841009
"unsloth": "Unsloth", "openai": "OpenAI"}
9851010

9861011

1012+
def engine_url(eng, cls=None, override_port=None):
1013+
"""Endpoint for an [[engine]] table: an explicit `url`, or the `host` /
1014+
`port` shorthand (host defaults to 127.0.0.1, port to the adapter's own
1015+
default). `override_port` re-points a URL at another port without dropping
1016+
its host or path (discovery)."""
1017+
url = eng.get("url") or ""
1018+
host = eng.get("host") or None
1019+
port = eng.get("port")
1020+
default_port = getattr(cls, "default_port", None) if cls else None
1021+
1022+
def netloc(u, h, p):
1023+
parts = urllib.request.urlparse(u)
1024+
return "%s:%s" % (h or parts.hostname or "127.0.0.1", p)
1025+
1026+
if override_port is not None:
1027+
port = override_port
1028+
if port is not None:
1029+
url = (urllib.request.urlparse(url)._replace(
1030+
netloc=netloc(url, host, port)).geturl()
1031+
if url else "http://%s:%s" % (host or "127.0.0.1", port))
1032+
elif host:
1033+
if url:
1034+
p = urllib.request.urlparse(url)
1035+
url = p._replace(
1036+
netloc=netloc(url, host, p.port or default_port)).geturl()
1037+
elif default_port:
1038+
url = "http://%s:%s" % (host, default_port)
1039+
return url
1040+
1041+
9871042
def make_adapter(eng, port=None):
9881043
"""Build an adapter from a config [[engine]] table (a plain dict of
9891044
type/url/...); `port` overrides the table's url port (discovery)."""
9901045
etype = eng.get("type") or "custom"
9911046
cls = ADAPTERS.get(etype, CustomAdapter)
992-
url = eng.get("url") or ""
993-
if port is not None and url:
994-
p = urllib.request.urlparse(url)
995-
url = p._replace(netloc=(p.hostname or "127.0.0.1") + ":" + str(port)
996-
).geturl()
1047+
url = engine_url(eng, cls, override_port=port)
9971048
a = cls(url=url or None, key=eng.get("name"), label=eng.get("name"),
9981049
parent=eng.get("parent"), api_key_env=eng.get("api_key_env"))
9991050
a.optional = bool(eng.get("optional")) # down is normal: no alert (speculum.toml)
@@ -1022,10 +1073,11 @@ class Scheduler:
10221073

10231074
def __init__(self, adapters, discovery_enabled=True, claimed_ports=None,
10241075
on_result=None, on_event=None, on_discover=None,
1025-
discovery_hosts=None):
1076+
discovery_hosts=None, discovery_ports=None):
10261077
self.discovery_enabled = discovery_enabled
10271078
self.claimed_ports = claimed_ports
10281079
self.discovery_hosts = list(discovery_hosts or ())
1080+
self.discovery_ports = list(discovery_ports or ())
10291081
self.on_result = on_result
10301082
self.on_event = on_event
10311083
self.on_discover = on_discover
@@ -1072,7 +1124,8 @@ def _claimed(self):
10721124
def discovery(self):
10731125
claimed = self._claimed()
10741126
found = []
1075-
for host, port, types in discovery_targets(self.discovery_hosts):
1127+
for host, port, types in discovery_targets(self.discovery_hosts,
1128+
self.discovery_ports):
10761129
if (host, port) in claimed:
10771130
continue
10781131
base = "http://%s:%d" % (host, port)

‎collector/speculum.py‎

Lines changed: 35 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -702,7 +702,7 @@ def logs_thread():
702702
e = S.engine("ninfer", model, "ninfer", 262144)
703703
try:
704704
# /logs/stream/<model> is a plain text tail, not SSE frames
705-
req = urllib.request.Request("http://127.0.0.1:9090/logs/stream/%s" % model,
705+
req = urllib.request.Request(LLAMA_SWAP + "/logs/stream/%s" % model,
706706
headers={"Accept": "text/plain"})
707707
r = urllib.request.urlopen(req, timeout=PROBE_SLOW)
708708
while True:
@@ -1408,13 +1408,35 @@ def _port_of(u):
14081408
return None
14091409

14101410

1411+
def endpoint_url(tbl, default):
1412+
"""The base URL for a built-in thread from its config table: `url`, or the
1413+
`host` / `port` shorthand on top of the default (e.g. llama-swap moved off
1414+
:9090). Always returned without a trailing slash."""
1415+
tbl = tbl or {}
1416+
url = tbl.get("url") or default
1417+
host, port = tbl.get("host"), tbl.get("port")
1418+
if port is not None or host:
1419+
p = urllib.request.urlparse(url)
1420+
url = p._replace(netloc="%s:%s" % (
1421+
host or p.hostname or "127.0.0.1", port if port is not None else p.port
1422+
)).geturl()
1423+
return url.rstrip("/")
1424+
1425+
14111426
def claimed_ports():
1412-
"""Ports the discovery probes must skip: llama-swap, Strata when up,
1413-
and the backend of any running entry (e.g. the NInfer proxy port)."""
1414-
ports = {9090}
1427+
"""Ports the discovery probes must skip: llama-swap's port, Strata's while
1428+
it is up, and the backend of any running entry (e.g. the NInfer proxy
1429+
port). Llama-swap is always skipped so its port is never mistaken for an
1430+
engine; Strata's port is released for discovery when Strata is down."""
1431+
ports = set()
1432+
p = _port_of(LLAMA_SWAP)
1433+
if p:
1434+
ports.add(p)
14151435
with lock:
14161436
if S.strata.get("up"):
1417-
ports.add(8080)
1437+
p = _port_of(STRATA)
1438+
if p:
1439+
ports.add(p)
14181440
for r in S.running or []:
14191441
p = _port_of(r.get("proxy") or "")
14201442
if p:
@@ -1471,8 +1493,10 @@ def start_scheduler(cfg):
14711493
adapters,
14721494
discovery_enabled=bool(disc.get("enabled", True)),
14731495
# localhost is always probed; `targets` adds remote hosts on the same
1474-
# port list, for a collector running away from the inference box.
1496+
# port list, for a collector running away from the inference box, and
1497+
# `ports` adds non-default ports for engines off their stock port.
14751498
discovery_hosts=disc.get("targets") or (),
1499+
discovery_ports=disc.get("ports") or (),
14761500
claimed_ports=claimed_ports,
14771501
on_result=on_engine_result,
14781502
on_event=lambda lv, m: push_event(lv, m),
@@ -1484,7 +1508,7 @@ def start_scheduler(cfg):
14841508

14851509

14861510
def main():
1487-
global PORT, HIST
1511+
global PORT, HIST, LLAMA_SWAP, STRATA
14881512
if sys.stdout is None: # pythonw / no console: a print would crash
14891513
sys.stdout = sys.stderr = open(os.devnull, "w")
14901514
ap = argparse.ArgumentParser()
@@ -1504,6 +1528,10 @@ def main():
15041528
elif cfg.get("server", {}).get("port"):
15051529
PORT = int(cfg["server"]["port"])
15061530
host = cfg.get("server", {}).get("host") or HOST
1531+
# Built-in threads can be moved off their stock ports too. NInfer follows
1532+
# llama-swap's /running proxy, so it needs no separate port.
1533+
LLAMA_SWAP = endpoint_url(cfg.get("llama_swap"), "http://127.0.0.1:9090")
1534+
STRATA = endpoint_url(cfg.get("strata"), "http://127.0.0.1:8080")
15071535

15081536
# SQLite history. retention_days 0 -> no-op object.
15091537
retention = int((cfg.get("history") or {}).get("retention_days", 30) or 0)

‎collector/test_config.py‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
#!/usr/bin/env python3
2+
"""Config tests: per-engine host/port keys survive cleaning, the built-in
3+
endpoint tables have defaults, and a TOML file overrides them."""
4+
5+
import importlib.util
6+
import tempfile
7+
import unittest
8+
from pathlib import Path
9+
10+
import config
11+
12+
13+
class TestCleanEngine(unittest.TestCase):
14+
def test_host_and_port_kept(self):
15+
e = config._clean_engine(
16+
{"type": "llamacpp", "host": "10.0.0.9", "port": 8081,
17+
"url": "http://127.0.0.1:8080", "unknown": "drop me"})
18+
self.assertEqual(e, {"type": "llamacpp", "host": "10.0.0.9",
19+
"port": 8081, "url": "http://127.0.0.1:8080"})
20+
21+
def test_defaults_carry_endpoint_tables(self):
22+
self.assertEqual(config.DEFAULTS["llama_swap"]["url"],
23+
"http://127.0.0.1:9090")
24+
self.assertEqual(config.DEFAULTS["strata"]["url"],
25+
"http://127.0.0.1:8080")
26+
self.assertEqual(config.DEFAULTS["discovery"]["ports"], [])
27+
28+
29+
@unittest.skipUnless(importlib.util.find_spec("tomllib"),
30+
"tomllib needs Python 3.11+")
31+
class TestLoadToml(unittest.TestCase):
32+
def _load(self, text):
33+
with tempfile.TemporaryDirectory() as d:
34+
p = Path(d) / "speculum.toml"
35+
p.write_text(text, encoding="utf-8")
36+
return config.load(str(p))[0]
37+
38+
def test_engine_port_survives(self):
39+
cfg = self._load(
40+
'[[engine]]\nname = "llama"\ntype = "llamacpp"\nport = 8081\n')
41+
self.assertEqual(cfg["engine"][0]["port"], 8081)
42+
43+
def test_discovery_ports_and_builtin_urls(self):
44+
cfg = self._load(
45+
'[discovery]\nports = [8081, 9091]\n'
46+
'[llama_swap]\nport = 9091\n'
47+
'[strata]\nurl = "http://127.0.0.1:8090"\n')
48+
self.assertEqual(cfg["discovery"]["ports"], [8081, 9091])
49+
self.assertEqual(cfg["llama_swap"]["port"], 9091)
50+
self.assertEqual(cfg["strata"]["url"], "http://127.0.0.1:8090")
51+
52+
53+
if __name__ == "__main__":
54+
unittest.main(verbosity=2)

0 commit comments

Comments
 (0)