You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(api-proxy): offer two priced models instead of one
The README claimed a second model was "more config, not code" without
showing it. Run nginx.conf.template twice with a different MODEL, give
each its own runners.json entry, app id, and price, and let the client
pick with --app. FLUX.1 schnell is cheaper than SD3 medium, so the two
prices differ for a reason a caller can see.
Each entry keeps its own capacity: the work happens upstream, so the
capabilities do not contend.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SD4o2Evku53Nb6zfgAdWxV
|[`hello-world`](./hello-world)| The simplest app: one request, one response | dynamic | single-shot | HTTP (JSON) | fixed |
43
-
|[`tiles`](./tiles)| Capacity fan-out — one call per tile | dynamic | single-shot | HTTP (base64 PNG) | fixed |
44
-
|[`api-proxy`](./api-proxy)| Pass calls through to a hosted API — the operator holds the key, callers pay per call| static | single-shot | HTTP (JPEG bytes) | fixed |
45
-
|[`echo`](./echo)| Realtime video, transformed and echoed back | dynamic | persistent | trickle | hour |
|[`hello-world`](./hello-world)| The simplest app: one request, one response | dynamic | single-shot | HTTP (JSON) | fixed |
43
+
|[`tiles`](./tiles)| Capacity fan-out — one call per tile | dynamic | single-shot | HTTP (base64 PNG) | fixed |
44
+
|[`api-proxy`](./api-proxy)| Pass calls through to hosted APIs — the operator holds the key, one capability per model| static | single-shot | HTTP (JPEG bytes) | fixed |
45
+
|[`echo`](./echo)| Realtime video, transformed and echoed back | dynamic | persistent | trickle | hour |
|[`realtime-transcription`](./realtime-transcription)| Audio up, transcripts back, on one socket | dynamic | persistent | WebSocket | hour |
48
48
49
49
Start with `hello-world` (the smallest end-to-end path); the others each layer on one new idea. More will follow, including a full example that exercises every feature. Each is self-contained and runs **offchain** (free, no wallet); most also run **on-chain** (paid) — see each README.
The live runner can also **pass calls through to an API that runs somewhere else** — here the **Hugging Face text-to-image inference API**. This example's runner is a **stock nginx**: [nginx.conf.template](nginx.conf.template) forwards each call to one pinned model URL and injects the operator's token. There is **no app code at all** — the orchestrator operator offers a hosted model ([Stable Diffusion 3 medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers)by default) as a paid capability with two config files.
3
+
The live runner can also **pass calls through to an API that runs somewhere else** — here the **Hugging Face text-to-image inference API**. This example's runner is a **stock nginx**: [nginx.conf.template](nginx.conf.template) forwards each call to one pinned model URL and injects the operator's token. There is **no app code at all** — the orchestrator operator offers two hosted models ([Stable Diffusion 3 medium](https://huggingface.co/stabilityai/stable-diffusion-3-medium-diffusers)and [FLUX.1 schnell](https://huggingface.co/black-forest-labs/FLUX.1-schnell)) as **two separately priced capabilities**, with nothing but config.
| Registration | static (orchestrator config + health poll) |
10
+
| Transport | HTTP (HF payload in, JPEG bytes out) |
11
+
| Pricing | fixed, per capability (0.0001 and 0.00004 USD per image) |
12
+
| Port | 8989 (each nginx service) |
13
13
14
14
Prerequisites (Docker, `uv`, and the [`livepeer-gateway` SDK](https://pypi.org/project/livepeer-gateway/)) and the shared on-chain/payment setup live in the [repo README](../README.md). The demo upstream additionally needs a **Hugging Face API token** (`HF_TOKEN`, from [huggingface.co → settings → tokens](https://huggingface.co/settings/tokens)) with inference-provider credits.
15
15
16
16
## How it's wired
17
17
18
-
The app is attached as a **static runner**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price — and health-polls `/health` (an nginx `return 200`). The `/proxy` location proxies to the pinned model URL (`MODEL` in [compose.yml](compose.yml)) with `Authorization: Bearer <HF_TOKEN>` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": "<prompt>"}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls.
18
+
The apps are attached as **static runners**: the orchestrator reads [runners.json](runners.json) via `-liveRunnerConfig` — app id, runner URL, single-shot mode, and the fixed price, one entry per capability — and health-polls each `/health` (an nginx `return 200`). The `/proxy` location proxies to the pinned model URL (`MODEL` in [compose.yml](compose.yml)) with `Authorization: Bearer <HF_TOKEN>` added. The caller's body is the [Hugging Face text-to-image payload](https://huggingface.co/docs/inference-providers/tasks/text-to-image) forwarded verbatim — `{"inputs": "<prompt>"}` — and the image comes back as **raw JPEG bytes**. The client calls it with `runner_selector` → `call_runner` ([client.py](client.py)) — discover, then one **single-shot** call per image, reading the bytes from `result.content`; the orchestrator reserves a session per call and releases it when the response returns. Grep `# Livepeer:` in client.py to see the exact calls.
19
+
20
+
**One service, one model, one capability.**`compose.yml` runs the same `nginx.conf.template` twice with a different `MODEL`, and `runners.json` gives each its own app id and price. `--app` on the client picks which one to call.
19
21
20
22
## Offering an API as a capability — what this shows
21
23
22
-
Everything is operator-side config. `runners.json` names the capability and sets the**fixed per-image price**; the nginx config pins the model URL and holds the credential (`HF_TOKEN`, from `.env`). The pinned URL is also the security model: the operator's credential can only be spent on exactly the offered model. The config pins the method to `POST` and drops the caller's query string too, so the body is the only thing a caller controls: they choose nothing but the prompt, and never see an API key. They discover the capability and pay **per image through Livepeer**, while the operator pays the upstream and prices above the per-image upstream cost.
24
+
Everything is operator-side config. `runners.json` names each capability and sets its**fixed per-image price**; the nginx config pins the model URL and holds the credential (`HF_TOKEN`, from `.env`). The pinned URL is also the security model: the operator's credential can only be spent on exactly the models offered. The config pins the method to `POST` and drops the caller's query string too, so the body is the only thing a caller controls: they choose nothing but the prompt, and never see an API key. They discover a capability and pay **per image through Livepeer**, while the operator pays the upstream and prices above the per-image upstream cost.
23
25
24
-
The app id names the model, not the proxy, because that is what callers discover: they match it exactly, so it has to say what they get. Swapping `MODEL` means renaming the app id with it.
26
+
**A capability is a product, not a parameter.** FLUX.1 schnell is faster and cheaper than SD3 medium, so it is its own app id at its own price rather than a flag on one shared endpoint. Callers pick between them the same way they pick between orchestrators: discovery filters on `app`, matched exactly, so the app id has to say what you get. Swapping a `MODEL` means renaming its app id with it, and offering a third model is the same change once more — one `runners.json` entry, one nginx service.
25
27
26
-
Offering a second model is more config, not code: one more `runners.json` entry (its own app id and price) plus one more nginx service with a different `MODEL`.
28
+
**Each entry carries its own capacity.** That is the right shape here: the two capabilities do not contend, because the work happens upstream at Hugging Face and nginx is only forwarding bytes. Runners that genuinely share a resource — several models resident on one GPU — are a different problem, since the orchestrator has no way to know that two registrations sit on the same card ([go-livepeer#4015](https://github.com/livepeer/go-livepeer/issues/4015)).
27
29
28
30
**Fixed pricing** is the natural fit: one call is one bounded unit of work, so the runner bills one flat price per call instead of metering time.
29
31
30
-
> [!NOTE]
31
-
> Registration can also be **dynamic**: an operator tool can `register_runner` several API endpoints at runtime, each as its own priced capability, without touching the orchestrator config. See [livepeer/api-proxy](https://github.com/livepeer/api-proxy) for an example of dynamic endpoint registration, with key storage and request stats for orchestrator operators.
32
+
## Pinned here, dynamic in livepeer/api-proxy
33
+
34
+
Both capabilities here are fixed at deploy time: two nginx services and two `runners.json` entries, changed by editing config and restarting. That is the whole point of a **static** runner, and it is the right trade when the offering is stable.
35
+
36
+
Registration can also be **dynamic**. [livepeer/api-proxy](https://github.com/livepeer/api-proxy) is one process that serves many endpoints, adding each at runtime with a CLI or a dashboard — one `register_runner` per endpoint, each its own priced capability, with no orchestrator config to touch and no restart. It also stores the upstream keys encrypted and reports per-endpoint request stats, which is what an operator running more than a handful of these actually needs.
37
+
38
+
Same idea, opposite ends of the same axis: pin two and read the config, or run one process and enable endpoints as you go.
32
39
33
40
## Run offchain (free)
34
41
35
42
```sh
36
43
cp .env.example .env # fill in HF_TOKEN; ignore the on-chain block
curl -sk https://localhost:8935/discovery | jq '.[].runners[] | {app, price_info}'#both capabilities, each with its price
39
46
uv run client.py --prompt "a watercolor painting of a llama writing code"
47
+
uv run client.py --app livepeer-example/flux-1-schnell \
48
+
--prompt "a watercolor painting of a llama writing code" --output flux-out.jpg
40
49
docker compose down
41
50
```
42
51
43
-
`compose.yml` brings up an orchestrator (`-useLiveRunners -liveRunnerConfig`) and the nginx runner. The client sends one prompt through the orchestrator and writes `api-proxy-out.jpg`.
52
+
`compose.yml` brings up an orchestrator (`-useLiveRunners -liveRunnerConfig`) and the two nginx runners. The client sends one prompt through the orchestrator per call and writes the image; `--app` chooses the model and `--output` keeps the two results apart.
44
53
45
54
## Run on-chain (paid)
46
55
47
-
Layer `compose.onchain.yml` to run the orchestrator on-chain with a remote signer paying each call — one fixed payment per image, at the price `runners.json` advertises. For the required RPC and wallets see [On-chain (paid) setup](../README.md#on-chain-paid-setup) in the repo README.
56
+
Layer `compose.onchain.yml` to run the orchestrator on-chain with a remote signer paying each call — one fixed payment per image, at the price `runners.json` advertises for the capability called. For the required RPC and wallets see [On-chain (paid) setup](../README.md#on-chain-paid-setup) in the repo README.
48
57
49
58
```sh
50
59
cp .env.example .env # fill in HF_TOKEN, RPC, network, keystore paths, accounts
@@ -55,4 +64,4 @@ uv run client.py --prompt "a watercolor painting of a llama writing code" \
55
64
docker compose -f compose.yml -f compose.onchain.yml down
56
65
```
57
66
58
-
Each call is one paid single-shot session — the orchestrator reserves it, takes one fixed payment, and releases it when the response returns.
67
+
Each call is one paid single-shot session — the orchestrator reserves it, takes one fixed payment, and releases it when the response returns. Call the cheaper capability with `--app livepeer-example/flux-1-schnell` and the payment is smaller, which is the whole reason each model is its own app. The signer's `MAX_PRICE_PER_UNIT` is a single cap across capabilities, so it has to clear the **highest** price in `runners.json`.
0 commit comments