Skip to content

Commit d023a79

Browse files
mpstatonmpstaton
committed
docs(deployment): add DEPLOYMENT.md, link from README
Standing reference for the live Railway deployment — services, volumes, env vars, DNS, redeploy commands, and every real gotcha hit setting it up (Railway CLI dot-path bug, volume creation panic, nats-server config requirements, Railpack turborepo auto-detection, Docker ARG empty-string fallback, Module Federation cross-origin assetPrefix, PORT auto-injection, wrong default flow on first sign-in). Written now, close to the work, rather than reconstructed later from memory. Also surfaced a previously-undiscovered naming mismatch while writing this: .env.example / local .env define JINA_AI_API_KEY, but services/content-ingest/src/jina.ts actually reads JINA_API_KEY (no "AI") — degrades gracefully to Jina's free tier, so nothing visibly broke, but the deployed content-ingest has been running on free-tier rate limits as a result. Logged under DEPLOYMENT.md's Known gaps, not fixed here. Files changed: - DEPLOYMENT.md (new) - README.md
1 parent 8fb8c12 commit d023a79

2 files changed

Lines changed: 259 additions & 0 deletions

File tree

‎DEPLOYMENT.md‎

Lines changed: 248 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,248 @@
1+
# Deployment
2+
3+
augment-it's single-tenant humain-vc instance runs on **Railway**, not the
4+
DigitalOcean droplet originally prepped for it (see [Why Railway, not
5+
DO](#why-railway-not-do)). This doc is the standing reference for how it's
6+
deployed; the narrative of *how it got this way* — including every bug hit
7+
along the way — lives in
8+
[`context-v/plans/Build-Order-Humain-VC-Unlock-Flow.md`](context-v/plans/Build-Order-Humain-VC-Unlock-Flow.md)
9+
(Steps 9–10) and the [changelog](changelog/2026-07-09_01_Augment-It-Deployed-Railway-Not-DigitalOcean-Custom-Domain-Live.md).
10+
11+
## Live URLs
12+
13+
| URL | Service | Purpose |
14+
|---|---|---|
15+
| `https://augment.didi.sh` | `shell` | The app itself — what users visit |
16+
| `wss://ws.augment.didi.sh/ws` | `workspace-service` | WebSocket endpoint every remote connects to directly |
17+
| `https://strategy-curator-production.up.railway.app/remoteEntry.js` | `strategy-curator` | Federated remote (static JS asset, not user-facing) |
18+
| `https://chat-production-3378.up.railway.app/remoteEntry.js` | `chat` | Federated remote (static JS asset, not user-facing) |
19+
20+
Both `shell` and `workspace-service` **must** stay on `*.didi.sh` — the
21+
`didi_session` cookie `id.didi.sh` issues is scoped to `Domain=.didi.sh`,
22+
and every federated remote's WS connections go to `workspace-service`
23+
directly. `strategy-curator` and `chat` are just static JS hosts loaded
24+
cross-origin into the shell's page; they don't need to share the cookie
25+
domain themselves.
26+
27+
## Why Railway, not DO
28+
29+
A DigitalOcean droplet (`167.172.42.247`) was prepped on 2026-07-06 as the
30+
original target. Re-checked its live numbers three days later, right before
31+
deploying, and found ~112MB free RAM (before running any of our own
32+
services) and a leftover `coolify-proxy` container still squatting on ports
33+
80/443 — "prepped and ready" had drifted. Given very few users and no prior
34+
DO ops investment (vs. real Fly.io experience from `id-didi-sh` deployed the
35+
same week), switched to Railway instead. Railway's multi-service-project
36+
model is also the closest 1:1 match to this repo's actual docker-compose
37+
shape. The droplet is still paid-for and untouched — a decision on whether
38+
to decommission it is still open.
39+
40+
## The 8 services
41+
42+
One Railway project (`augment-it`), workspace **The Lossless Group**, one
43+
environment (`production`).
44+
45+
| Service | Source | Build | Port | Public domain |
46+
|---|---|---|---|---|
47+
| `nats` | Docker image `nats:2.10-alpine` | image (custom start command) | 4222 / 8222 | none (private only) |
48+
| `workspace-service` | `services/workspace/Dockerfile` | Dockerfile | 3001 | `ws.augment.didi.sh` |
49+
| `record-surrealdb-resolver` | `services/record-surrealdb-resolver/Dockerfile` | Dockerfile | — | none |
50+
| `content-ingest` | `services/content-ingest/Dockerfile` | Dockerfile | — | none |
51+
| `prompt-runner` | `services/prompt-runner/Dockerfile` | Dockerfile | — | none |
52+
| `shell` | `shell/Dockerfile` | Dockerfile | 3100 | `augment.didi.sh` |
53+
| `strategy-curator` | `apps/strategy-curator/Dockerfile` | Dockerfile | 3017 | Railway-generated |
54+
| `chat` | `apps/chat/Dockerfile` | Dockerfile | 3006 | Railway-generated |
55+
56+
**Every service is Dockerfile-built, including the three frontends** —
57+
Railway's Railpack auto-builder is not used, deliberately (see
58+
[Gotchas](#gotchas-hit-worth-knowing-before-touching-this-again)). The four
59+
backend services each have `source.rootDirectory` set to their own
60+
`/services/<name>`; the three frontends do **not** have a `rootDirectory`
61+
set (they need the full pnpm workspace context — `shared monorepo`
62+
pattern), and their Dockerfiles `COPY` the whole repo before running
63+
`pnpm --filter <pkg> build`.
64+
65+
**Only two remotes are actually wired for this deploy**: `strategyCurator`
66+
and `chat`. The other twelve remotes `shell/rsbuild.config.ts` knows about
67+
(`recordCollector`, `promptTemplateManager`, …) stay hardcoded to
68+
`localhost` — they belong to flows this single-tenant instance doesn't use.
69+
Module Federation remotes are lazy-loaded in the sense that matters here:
70+
nobody on this instance ever navigates to them, so their broken state is
71+
harmless (though the browser DOES eagerly probe every declared remote's
72+
`remoteEntry.js` on shell load, so their failures show up as console noise
73+
— cosmetic, not functional).
74+
75+
## Volumes
76+
77+
Two Railway Volumes, deliberately **not shared** — confirmed via Railway's
78+
own docs and support that a volume is strictly single-service:
79+
80+
- **`content-ingest`** owns `/clients` — the real corpus filesystem,
81+
read-write. This is the humain-vc corpus's actual home.
82+
- **`workspace-service`** owns its own tiny `/data` volume, containing
83+
`sessions.json` (WS session tokens) and a **self-seeded**
84+
`clients/humain-vc/.env` stub (`DEFAULT_DOMAIN_TYPE=thesis`, nothing
85+
else). Written fresh on every container boot via `deploy.startCommand`
86+
(see below) rather than uploading the real `clients/humain-vc/.env` —
87+
that file also holds unrelated Decile Hub credentials workspace-service
88+
has no business touching.
89+
90+
`workspace-service`'s custom start command:
91+
92+
```sh
93+
sh -c 'mkdir -p /data/clients/humain-vc && echo DEFAULT_DOMAIN_TYPE=thesis > /data/clients/humain-vc/.env && npm start'
94+
```
95+
96+
`nats`'s custom start command (see [Gotchas](#gotchas-hit-worth-knowing-before-touching-this-again) for why it's shaped this way):
97+
98+
```sh
99+
sh -c "echo port: 4222 > /tmp/nats.conf && echo http_port: 8222 >> /tmp/nats.conf && echo max_payload: 48MB >> /tmp/nats.conf && nats-server -c /tmp/nats.conf"
100+
```
101+
102+
## Environment variables
103+
104+
Names only — see Railway's dashboard or `railway variable list --service
105+
<name> --json` for values (never printed to a transcript; see the
106+
[Gotchas](#gotchas-hit-worth-knowing-before-touching-this-again) note on
107+
credential handling).
108+
109+
| Service | Variables |
110+
|---|---|
111+
| `workspace-service` | `NATS_URL` (`nats://${{nats.RAILWAY_PRIVATE_DOMAIN}}:4222`), `CLIENTS_ROOT=/data/clients`, `SESSION_STORE_PATH=/data/sessions.json`, `ID_JWKS_URL`, `ID_ISSUER`, `DIDI_AUTH=required`, `REQUIRED_ORG_ID=humain.vc`, `ACTIVE_CLIENT_ID=humain-vc`, `PORT=3001` (see gotcha below) |
112+
| `record-surrealdb-resolver` | `NATS_URL`, `SURREAL_URL`, `SURREAL_NS`, `SURREAL_DB`, `SURREAL_USER`, `SURREAL_PASS` |
113+
| `content-ingest` | `NATS_URL`, `CLIENTS_ROOT=/clients`, `JINA_API_KEY` (**not currently set** — see [Known gaps](#known-gaps)) |
114+
| `prompt-runner` | `NATS_URL`, `ANTHROPIC_API_KEY` |
115+
| `shell` | `PUBLIC_WS_URL=wss://ws.augment.didi.sh/ws`, `PUBLIC_ID_BASE=https://id.didi.sh`, `PUBLIC_STRATEGY_CURATOR_REMOTE`, `PUBLIC_CHAT_REMOTE` (all build-time — baked in via Docker `ARG`/`ENV`, not read at runtime) |
116+
| `strategy-curator` | `PUBLIC_WS_URL`, `PUBLIC_STRATEGY_CURATOR_ASSET_PREFIX` (build-time) |
117+
| `chat` | `PUBLIC_WS_URL`, `PUBLIC_CHAT_ASSET_PREFIX` (build-time) |
118+
119+
`PUBLIC_*` vars on the three frontends only take effect on the **next
120+
build** — changing one requires `railway redeploy --service <name>
121+
--from-source`, not just a restart.
122+
123+
## DNS
124+
125+
Two custom domains at Vercel DNS (where `didi.sh` is registered), each
126+
needing a CNAME + a one-time TXT ownership-verification record:
127+
128+
| Host | Type | Points to |
129+
|---|---|---|
130+
| `augment` | CNAME | Railway-issued target (see `railway domain status augment.didi.sh`) |
131+
| `ws.augment` | CNAME | Railway-issued target (see `railway domain status ws.augment.didi.sh`) |
132+
133+
`id-didi-sh`'s production CORS allowlist (`config/runtime.exs`,
134+
`:identity, cors_origins:`) includes `https://augment.didi.sh` — it was
135+
**empty** in prod before this, meaning every cross-origin browser call to
136+
`id.didi.sh` had been silently failing since it first deployed. Add each
137+
new `*.didi.sh` consumer (decks, memos, …) to that list as it goes live.
138+
139+
## Deploying / redeploying
140+
141+
```bash
142+
# One-time context setup (per shell session)
143+
export RAILWAY_CALLER="skill:use-railway@1.3.4"
144+
export RAILWAY_AGENT_SESSION="<stable-id-for-this-session>"
145+
146+
# Redeploy one service from its latest pushed commit
147+
railway redeploy --service <name> --from-source --yes
148+
149+
# Check status — NEVER trust "queued", poll until terminal
150+
railway deployment list --service <name> --environment production --json
151+
152+
# Tail logs
153+
railway logs --service <name> --lines 100 --json
154+
155+
# Change config (dot-path form is unreliable in the CLI version used to
156+
# set this up — see Gotchas. Use the JSON-patch form:)
157+
railway environment edit --json <<'JSON'
158+
{"services":{"<service-id>":{"variables":{"KEY":{"value":"..."}}}}}
159+
JSON
160+
```
161+
162+
Service IDs, not names, are required for the JSON-patch form — resolve via
163+
`railway environment config --json` (dumps every service's current config,
164+
keyed by ID).
165+
166+
## Gotchas hit, worth knowing before touching this again
167+
168+
- **Railway CLI's `environment edit --service-config` (dot-path form)
169+
silently no-ops** in the version used to set this up (5.25.1) —
170+
`{"committed":false,"message":"No changes to apply"}` regardless of the
171+
value. The **JSON-patch form** works reliably; used for everything.
172+
- **`railway volume add` panics** (Rust `unwrap()` on `None`) in the same
173+
CLI version. Volumes were created via a direct GraphQL `volumeCreate`
174+
mutation instead (`scripts/railway-api.sh` from the `use-railway` skill,
175+
or any GraphQL client against `https://backboard.railway.com/graphql/v2`).
176+
- **`nats-server` does not accept `-max_payload` as a CLI flag** — this
177+
repo's own `nats.conf` already knew that from months back (a bind-mount
178+
in local docker-compose exists for exactly this reason). Needed an
179+
inline-generated config file on Railway; the first attempt
180+
(`printf '...\n...\n'` with embedded newline escapes) got corrupted
181+
somewhere across the Railway CLI → GraphQL → container `sh -c` chain.
182+
The form that survives: one `echo` per line, no embedded `\n` at all.
183+
- **Railway's Railpack builder auto-detects a turborepo** (`turbo.json`
184+
exists at this repo's root) and unconditionally runs the root
185+
`package.json`'s `build` script (`turbo run build`) for any
186+
Railpack-built service, **ignoring any custom `buildCommand`
187+
override**. `turbo` was never an actually-installed binary in this
188+
repo. Fixed by giving all three frontends their own Dockerfiles instead
189+
of relying on Railpack at all.
190+
- **A Docker `ARG` that's declared but never passed resolves to an empty
191+
string, not `undefined`** — `?? default` in TypeScript doesn't catch
192+
it. Two real bugs from this: `shell/rsbuild.config.ts`'s remote-URL
193+
fallbacks had to change from `??` to `||`, and the same for the
194+
`PUBLIC_WS_URL` fallback in `shell`/`strategy-curator`/`chat`.
195+
- **Federated remotes need `output.assetPrefix`, not just
196+
`dev.assetPrefix`.** Missing it in production meant `chat`'s and
197+
`strategy-curator`'s async sub-chunks resolved as relative paths
198+
against the **shell's** origin instead of their own, 404ing into the
199+
shell's SPA-fallback HTML (`SyntaxError: Unexpected token '<'` — that
200+
HTML being `eval`'d as JS). Only reproduces cross-origin; local
201+
federation dev never surfaces it.
202+
- **Railway auto-injects its own `PORT`** (8080) for any service with a
203+
public domain. This silently mismatched `workspace-service`'s domain
204+
target port (3001, set explicitly when the domain was created) until
205+
`PORT=3001` was set as an explicit service variable.
206+
- **The shell defaulted to the wrong flow on first sign-in.**
207+
`FLOWS[0]` (`csvAugmentation` — Record Collector's flow, whose remotes
208+
are the twelve deliberately-unreachable ones) was the module-init-time
209+
default, resolved before `workspace.pinned` was known. A fresh sign-in
210+
on the pinned humain-vc instance landed there first, showing "remote
211+
exposes no mount function" as the very first thing a new user saw.
212+
Fixed with `activeFlow.applyPinnedDefault()`, applied once
213+
`workspace.pinned` resolves true, only when the user has never made an
214+
explicit flow choice.
215+
- **Don't print secret variable values into a transcript**, even when
216+
trying to redact — a `isSealed`-flag check isn't the same as actually
217+
not fetching the value. `railway variable set --stdin` (piped from a
218+
local `.env`, never echoed) is safe; reading values back for
219+
verification isn't — check `{"set": true}`-style confirmations only.
220+
221+
## Known gaps
222+
223+
- **`JINA_API_KEY` naming mismatch.** `.env.example` and this repo's local
224+
`.env` both define `JINA_AI_API_KEY`; `services/content-ingest/src/jina.ts`
225+
actually reads `process.env.JINA_API_KEY` (no "AI"). Degrades gracefully
226+
to Jina's free/no-auth tier when unset, so nothing has visibly broken —
227+
but the deployed `content-ingest` is running on the free tier's rate
228+
limits as a result. Either rename the var everywhere for consistency, or
229+
set `JINA_API_KEY` (the name the code actually reads) on Railway with the
230+
real key value.
231+
- **Corpus sync / backup.** The corpus lives on a single Railway Volume
232+
with no automated backup or sync-to-laptop story yet — the original plan
233+
assumed a DO box's filesystem an rclone cron job could reach directly,
234+
which doesn't translate to Railway's volume model. Needs a Railway-native
235+
redesign (most likely a periodic job inside `content-ingest` itself
236+
pushing to R2) before this is a durable setup.
237+
- **The DO droplet** (`167.172.42.247`) is unused but still provisioned and
238+
billed. Decommission-or-keep-as-spare is an open decision.
239+
- **No human sign-in has been load-tested** beyond the operator's own
240+
verification — Aneil and Linea have `org_owner` accounts seeded
241+
(production + local) but haven't yet completed a live dress-rehearsal
242+
session together.
243+
244+
## Related
245+
246+
- [`context-v/plans/Build-Order-Humain-VC-Unlock-Flow.md`](context-v/plans/Build-Order-Humain-VC-Unlock-Flow.md) — Steps 9–10, the full narrative
247+
- [`changelog/2026-07-09_01_Augment-It-Deployed-Railway-Not-DigitalOcean-Custom-Domain-Live.md`](changelog/2026-07-09_01_Augment-It-Deployed-Railway-Not-DigitalOcean-Custom-Domain-Live.md)
248+
- [`context-v/specs/Id-Didi-Sh-Identity-Service.md`](../context-v/specs/Id-Didi-Sh-Identity-Service.md) (ai-labs level) — the identity/cookie contract this deploy depends on

‎README.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,17 @@ pnpm preview # Preview the production build
135135

136136
The `scripts/dev.sh` script prints the full URL list on start.
137137

138+
## Deployment
139+
140+
The humain-vc single-tenant instance runs live on **Railway** at
141+
[`https://augment.didi.sh`](https://augment.didi.sh) — 8 services (NATS +
142+
5 backend microservices + 3 federated frontends), two persistent volumes,
143+
and a custom `*.didi.sh` domain (required for the shared `didi_session`
144+
cookie). See **[`DEPLOYMENT.md`](DEPLOYMENT.md)** for the full service
145+
list, environment variables, redeploy commands, and the real gotchas hit
146+
getting it there (Railway CLI quirks, Module Federation cross-origin
147+
asset resolution, `nats-server` config).
148+
138149
## Conventions
139150

140151
- **Branch tiers:** `development` → `main` → `master`. Parent on tier X → all submodules on tier X.

0 commit comments

Comments
 (0)