Skip to content

Commit 489e1bc

Browse files
authored
refactor: make worker startup the only flow compilation path (#672)
# refactor: make worker startup the only flow compilation path Implements #647 ahead of queue identity (#650) and per-step private queues (#651). ## Summary - Every flow worker now compiles or verifies its imported flow definition before it tracks its worker function or inserts its worker row. A failed compilation registers nothing and mutates no `worker_functions` row. - Removed public surfaces: `compileFlow()` and `compareFlowShapes()` from `@pgflow/dsl`, the ControlPlane edge function and its root export, the `pgflow compile` CLI command, the ControlPlane installer step, all `supabase/functions/pgflow/` fixtures, and the complete `FlowWorkerConfig.compilation` property (including `allowDataLoss`). - SQL: `pgflow.ensure_flow_compiled(flow_slug, shape)` keeps two arguments only. The new Atlas migration explicitly drops `pgflow.ensure_flow_compiled(text, jsonb, boolean)`, so production can no longer authorize destructive recompilation. - Local same-slug shape mismatch keeps destructive recompile (dev hot reload); production mismatch returns `mismatch` without touching data. Structural or unobserved semantic changes use a new concrete slug. - Kept: `extractFlowShape()`, `FlowShape` types, SQL `_compare_flow_shapes()`, `_create_flow_from_shape()`, `create_flow()`, `add_step()`, `delete_flow_and_data()`. - Docs rewritten for startup-only deployment: `/concepts/startup-compilation/` is canonical, legacy URLs redirect there, and the [pgflow 0.16.0 release notes](https://pgflow.dev/news/pgflow-0-16-0-startup-only-flow-compilation/) document the fenced breaking-upgrade procedure. - Changeset bumps the fixed group (`@pgflow/core`, `@pgflow/dsl`, `@pgflow/client`, `@pgflow/edge-worker`, `pgflow`) as minor. **Breaking:** the migration removes the old three-argument SQL signature — stop old workers before applying it, and upgrade the pgflow package set together. See "Remove manual flow compilation" in the update guide. ## Checks All run locally on this branch: - `pnpm nx verify-migrations core`, `pnpm nx gen-types core`, `pnpm nx verify-gen-types core --skip-nx-cache` — pass - `pnpm nx test:pgtap core` — 1348/1348 pass (incl. new two-arg signature test) - `pnpm nx run-many -t lint build --projects=cli,dsl,edge-worker,client` — pass - `pnpm nx test cli` / `test dsl` (337) / `test client` — pass - `pnpm nx test:unit edge-worker` (271) / `test:types edge-worker` / `test:integration edge-worker` (61/61) — pass - `pnpm nx e2e cli` / `e2e edge-worker` (12/12) / `e2e client` (47/47) — pass - `pnpm nx verify-exports edge-worker` / `client`, publish dry runs (npm/JSR) — pass - `pnpm nx build website` — 133 pages, starlight-links-validator pass - CLI help contains no `compile` command; packed exports contain no `ControlPlane`/`compileFlow`/`compareFlowShapes` Closes #647
1 parent 5118509 commit 489e1bc

91 files changed

Lines changed: 545 additions & 4683 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
'@pgflow/core': minor
3+
'@pgflow/dsl': minor
4+
'@pgflow/client': minor
5+
'@pgflow/edge-worker': minor
6+
'pgflow': minor
7+
---
8+
9+
Remove migration-based flow compilation. Flow workers now always compile or verify their imported definition before registration and polling.
10+
11+
This removes `compileFlow()`, `compareFlowShapes()`, `ControlPlane`, `pgflow compile`, `FlowWorkerConfig.compilation`, and production `allowDataLoss`. The database migration removes `pgflow.ensure_flow_compiled(text, jsonb, boolean)`, so stop old workers before applying it and upgrade the fixed pgflow package set together.

‎ARCHITECTURE_GUIDE.md‎

Lines changed: 8 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -123,13 +123,7 @@ const CompleteExample = new Flow<{ urls: string[] }>({
123123
export default CompleteExample;
124124
```
125125

126-
**Compilation**:
127-
```bash
128-
npx pgflow compile path/to/flow.ts
129-
# Generates migration with:
130-
# - SELECT pgflow.create_flow(...)
131-
# - SELECT pgflow.add_step(...) for each step
132-
```
126+
**Compilation**: Worker startup extracts the complete flow shape (`extractFlowShape()`) and PostgreSQL compiles or verifies it via `pgflow.ensure_flow_compiled(flow_slug, shape)` before any registration.
133127

134128
**Important**: See DSL package files for:
135129
- Type inference utilities: `ExtractFlowInput`, `ExtractFlowOutput`, `StepInput`, `StepOutput`
@@ -203,7 +197,10 @@ npx pgflow compile path/to/flow.ts
203197
**Mental model**: Execution engine. Doesn't know where tasks come from or overall workflow state.
204198

205199
**Worker Lifecycle**:
206-
1. `acknowledgeStart()` - Register worker with `workerId` in database
200+
1. `acknowledgeStart()`:
201+
- Compile or verify the imported flow shape (`ensureFlowCompiled`)
202+
- Track the worker function (`track_worker_function`)
203+
- Insert the worker row with `workerId` in the database
207204
2. Main loop:
208205
- `sendHeartbeat()` - Update status, check deprecation
209206
- If deprecated → exit gracefully
@@ -281,7 +278,6 @@ await worker.start();
281278

282279
**Commands**:
283280
- `pgflow install [--supabase-path <path>] [-y, --yes]` - Sets up pgflow
284-
- `pgflow compile <flow.ts> [--deno-json <path>] [--supabase-path <path>]` - Compiles flows to SQL
285281

286282
**See**: `/pkgs/cli/src/commands/` for implementation details
287283

@@ -330,12 +326,13 @@ await worker.start();
330326
**Why**: Prevents infinite waiting on steps that will never have tasks.
331327

332328
**How**:
333-
- DSL: Map steps compiled with `step_type => 'map'`
329+
- DSL: Map steps carry `stepType: 'map'` in the extracted shape; `_create_flow_from_shape` persists it as `step_type = 'map'`
334330
- SQL Core: `complete_task()` detects array parent completed with `[]`, sets dependent map `initial_tasks=0`
335331
- SQL Core: `cascade_complete_taskless_steps()` completes all `initial_tasks=0` steps in chain (max 50 iterations)
336332

337333
**See**:
338-
- DSL compilation: `/pkgs/dsl/src/compiler/compileFlow.ts`
334+
- Shape extraction: `/pkgs/dsl/src/flow-shape.ts`
335+
- SQL compiler: `/pkgs/core/schemas/0100_function_create_flow_from_shape.sql`
339336
- Cascade logic: `/pkgs/core/src/migrations/` (function: `cascade_complete_taskless_steps`)
340337

341338
### 3. Realtime Events (SQL Core + Client)

‎NOMENCLATURE_GUIDE.md‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -241,12 +241,10 @@ Step handlers use **asymmetric signatures** based on whether they have dependenc
241241
| Command | Description |
242242
| ---------------------------------- | ---------------------------------------- |
243243
| `npx pgflow@latest install` | Install pgflow in Supabase project |
244-
| `npx pgflow@latest compile <file>` | Compile TypeScript flow to SQL migration |
245244

246245
**Common Options**:
247246

248247
- `--supabase-path <path>` - Custom Supabase directory
249-
- `--deno-json <path>` - Custom deno.json config
250248
- `--yes` / `-y` - Skip confirmation prompts
251249

252250
## Client API

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -84,8 +84,8 @@ See how pgflow compares to [Trigger.dev](https://pgflow.dev/comparisons/trigger/
8484
## How it works
8585

8686
1. **Define workflows** using the TypeScript DSL
87-
2. **Compile** them to SQL migrations
88-
3. **Deploy** as Supabase Edge Functions
87+
2. **Deploy** as Supabase Edge Functions
88+
3. **Compile & verify** automatically in the database at worker startup
8989
4. **Trigger** from your app, SQL, or pg_cron
9090

9191
The execution engine handles scheduling, retries, and result aggregation automatically.
@@ -94,7 +94,7 @@ The execution engine handles scheduling, retries, and result aggregation automat
9494

9595
| Package | Version | Description |
9696
| ------------------------------------------ | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
97-
| [pgflow](./pkgs/cli/) | [![npm](https://img.shields.io/npm/v/pgflow)](https://www.npmjs.com/package/pgflow) | CLI for installing and compiling flows |
97+
| [pgflow](./pkgs/cli/) | [![npm](https://img.shields.io/npm/v/pgflow)](https://www.npmjs.com/package/pgflow) | CLI for installing pgflow into your Supabase project |
9898
| [@pgflow/core](./pkgs/core/) | [![npm](https://img.shields.io/npm/v/@pgflow/core)](https://www.npmjs.com/package/@pgflow/core) | SQL Core - foundational tables and functions |
9999
| [@pgflow/dsl](./pkgs/dsl/) | [![npm](https://img.shields.io/npm/v/@pgflow/dsl)](https://www.npmjs.com/package/@pgflow/dsl) | TypeScript DSL for defining flows with type inference |
100100
| [@pgflow/edge-worker](./pkgs/edge-worker/) | [![JSR](https://jsr.io/badges/@pgflow/edge-worker)](https://jsr.io/@pgflow/edge-worker) | Task queue worker for Supabase Edge Functions |

‎apps/demo/supabase/functions/pgflow/deno.json‎

Lines changed: 0 additions & 12 deletions
This file was deleted.

‎apps/demo/supabase/functions/pgflow/index.ts‎

Lines changed: 0 additions & 4 deletions
This file was deleted.

‎context7.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727
"Root map steps omit `array:`, dependent maps require it - Root maps process flow input directly without `array:` property (e.g., `new Flow<string[]>().map({ slug: 'process' }, ...)`); dependent maps must specify `array:` to indicate which step's output to process.",
2828
"Steps with the same dependencies run in parallel - pgflow automatically maximizes parallelism by running independent steps concurrently once their dependencies are satisfied.",
2929
"Handler functions receive context as second parameter - Root steps: `(flowInput, ctx)`, dependent steps: `(deps, ctx)`. Context provides platform resources like `ctx.sql`, `ctx.supabase`, `ctx.env`, `ctx.flowInput`, `ctx.stepTask`, and `ctx.rawMessage` without requiring global imports or connection setup.",
30-
"Flows must be compiled before execution - Run `npx pgflow@latest compile` to convert TypeScript flow definitions into SQL migrations that register the flow structure in the database.",
30+
"Flows are deployed by worker startup - A worker compiles or verifies its imported flow definition in the database before registration and polling; there is no separate compile command or migration.",
3131
"Registered flows are immutable - Once a flow is registered in the database, its structure cannot be modified; use versioning or deletion (development only) to make changes.",
3232
"Use camelCase for step slugs - Step slugs follow JavaScript naming conventions (e.g., `fetchData`, not `fetch_data`) since they become property names in TypeScript for accessing outputs.",
3333
"Name steps based on their primary purpose - Use nouns when downstream steps process the returned data (e.g., `website`, `userProfiles`); use verb-noun when the side effect is primary (e.g., `sendEmails`, `saveToDb`).",

‎pkgs/cli/README.md‎

Lines changed: 5 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,10 @@ The Command Line Interface for pgflow - a PostgreSQL-native workflow engine.
77
88
## Overview
99

10-
This package provides essential tools for setting up, managing, and deploying pgflow workflows in your Supabase environment. The CLI handles:
10+
This package provides essential tools for setting up pgflow in your Supabase project. The CLI handles:
1111

1212
- Installing pgflow in your Supabase project
13-
- Compiling TypeScript workflow definitions into SQL migrations
14-
- Managing workflow deployment and updates
13+
- Updating pgflow's project files (safe to re-run)
1514

1615
## Prerequisites
1716

@@ -59,26 +58,6 @@ The installer will:
5958
3. Configure environment variables for Edge Functions
6059
4. Guide you through applying migrations
6160

62-
### Compile Flow Definition
63-
64-
Convert a TypeScript flow definition into a SQL migration:
65-
66-
```bash
67-
npx pgflow@latest compile my_flow
68-
```
69-
70-
Options:
71-
72-
- `--supabase-path <path>` - Path to custom Supabase directory
73-
- `--control-plane-url <url>` - ControlPlane URL (default: `http://127.0.0.1:54321/functions/v1/pgflow`)
74-
75-
The compiler will:
76-
77-
1. Parse your TypeScript flow definition
78-
2. Extract step dependencies and configuration
79-
3. Generate SQL commands for database registration
80-
4. Create a timestamped migration file in your migrations folder
81-
8261
## Building
8362

8463
Run `nx build cli` to build the library.
@@ -91,6 +70,6 @@ Run `nx test cli` to execute the unit tests via [Vitest](https://vitest.dev/).
9170

9271
For detailed documentation, visit:
9372

94-
- [Installation Guide](https://pgflow.dev/getting-started/install-pgflow/)
95-
- [Compiling Flows](https://pgflow.dev/getting-started/compile-to-sql/)
96-
- [Running Flows](https://pgflow.dev/getting-started/run-flow/)
73+
- [Installation Guide](https://pgflow.dev/get-started/installation/)
74+
- [Startup Compilation](https://pgflow.dev/concepts/startup-compilation/)
75+
- [Running Flows](https://pgflow.dev/get-started/flows/run-flow/)

0 commit comments

Comments
 (0)