Skip to content

Commit 99bef82

Browse files
jumskiAgent
andauthored
docs: fence worker_functions row before deprecating workers in update guide (#670)
The production Update Deployed Flows guide told operators to deprecate old workers, deploy, and wait for cron. While the pgflow.worker_functions row stays enabled, the ensure_workers() cron (1s interval) can start another instance of the old deployed function between deprecation and deployment, because deprecated workers do not count as alive for the restart check. Rewrite the guide around a disable fence for one worker function: - record the current enabled value, then set enabled = false - deprecate its live pgflow.workers rows - verify polling stopped (heartbeat window) and in-flight work drained (step_tasks.status leaves 'started' for the deprecated workers) - deploy the replacement while the row stays disabled - restore the exact recorded enabled value, not an inferred one - confirm the replacement worker reports a current heartbeat Add failure guidance covering a worker that does not drain or does not restart, with inspection SQL (worker listing, stuck task detail, last_invoked_at, cron.job entry). Link the corrected sequence from the Worker Deprecation section of the Worker Management guide. Per-step, alias-version, and shared-queue rollout instructions stay with issues #651, #648, and #652; deliberately not documented here. Closes #654 Co-authored-by: Agent <agent@pgflow.dev>
1 parent 4ac87c9 commit 99bef82

2 files changed

Lines changed: 159 additions & 16 deletions

File tree

‎pkgs/website/src/content/docs/deploy/supabase/update-deployed-flows.mdx‎

Lines changed: 152 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Update Deployed Flows
3-
description: Safe workflow for deploying flow updates to production using worker deprecation
3+
description: Disable restarts, deprecate workers, drain, deploy, then re-enable to update deployed flows safely
44
sidebar:
55
order: 110
66
banner:
@@ -10,49 +10,186 @@ banner:
1010

1111
import { Aside, Steps } from "@astrojs/starlight/components";
1212

13-
When updating flows that are already deployed to production, use the deprecation workflow to prevent old and new worker versions from processing tasks simultaneously.
13+
When updating flows that are already deployed to production, follow this sequence to prevent old and new worker versions from processing tasks simultaneously. The order matters: the `ensure_workers()` cron restarts any function whose `worker_functions` row is `enabled`, so you must disable the row before you deprecate workers.
14+
15+
See [Worker Management](/deploy/worker-management/) for how worker registration, deprecation, and heartbeats work.
1416

1517
<Aside type="note" title="First Time Deployment?">
1618
If you haven't deployed to production yet, see [Deploy Your First Flow](/deploy/supabase/deploy-first-flow/) for initial setup instructions.
1719
</Aside>
1820

1921
## Update Workflow
2022

23+
Throughout, replace `your-worker-name` with your Edge Function name.
24+
2125
<Steps>
2226

23-
1. **Deprecate old workers**
27+
1. ### Record the current enabled state
28+
29+
```sql
30+
SELECT function_name, enabled
31+
FROM pgflow.worker_functions
32+
WHERE function_name = 'your-worker-name';
33+
```
34+
35+
Note the `enabled` value. You restore exactly this value in step 6, even if it is `false`.
36+
37+
2. ### Disable automatic restarts
38+
39+
```sql
40+
UPDATE pgflow.worker_functions
41+
SET enabled = false
42+
WHERE function_name = 'your-worker-name';
43+
```
44+
45+
While `enabled` is `false`, the cron stops pinging this function. Disable the row before deprecating workers, otherwise the cron restarts the old worker within a second of the deprecation.
46+
47+
3. ### Deprecate live workers
48+
2449
```sql
2550
UPDATE pgflow.workers
26-
SET deprecated_at = NOW()
51+
SET deprecated_at = now()
2752
WHERE function_name = 'your-worker-name'
2853
AND deprecated_at IS NULL;
2954
```
30-
This stops old workers from polling for new tasks while they finish current work.
3155

32-
2. **Deploy new version**
56+
Workers check their deprecation status every 5 seconds. A deprecated worker stops polling for new tasks and finishes the tasks it already claimed.
57+
58+
4. ### Wait for the drain to finish
59+
60+
A worker stopped polling when none of its function's workers has a current heartbeat:
61+
62+
```sql
63+
SELECT worker_id, deprecated_at, last_heartbeat_at
64+
FROM pgflow.workers
65+
WHERE function_name = 'your-worker-name'
66+
AND last_heartbeat_at > now() - interval '6 seconds';
67+
```
68+
69+
This returns no rows once every worker of the function stopped polling.
70+
71+
In-flight work finished when no task claimed by a deprecated worker is still running:
72+
73+
```sql
74+
SELECT run_id, step_slug, task_index, started_at
75+
FROM pgflow.step_tasks
76+
WHERE status = 'started'
77+
AND last_worker_id IN (
78+
SELECT worker_id
79+
FROM pgflow.workers
80+
WHERE function_name = 'your-worker-name'
81+
AND deprecated_at IS NOT NULL
82+
);
83+
```
84+
85+
This returns no rows when every claimed task reached a terminal state (`completed` or `failed`) or was requeued back to `queued`.
86+
87+
<Aside type="note">
88+
pgflow does not record completion callbacks as separate rows. The observable signal is the task row leaving `started`: the worker writes the final task state only after the handler and its completion callbacks finish.
89+
</Aside>
90+
91+
Repeat both queries until they return no rows. The drain takes as long as the longest in-flight task. If the function had no live workers, both queries return no rows immediately.
92+
93+
5. ### Deploy the new version
94+
3395
```bash frame="none"
3496
npx supabase functions deploy your-worker-name
3597
```
3698

37-
3. **Wait for new worker to start**
99+
The function row stays disabled during deployment, so nothing restarts the worker while the new code deploys.
100+
101+
6. ### Restore the recorded enabled state
38102

39-
The pgflow cron automatically starts the new worker within seconds.
103+
```sql "true"
104+
UPDATE pgflow.worker_functions
105+
SET enabled = true
106+
WHERE function_name = 'your-worker-name';
107+
```
40108

41-
See [Worker Management](/deploy/worker-management/) for details.
109+
Set the value you recorded in step 1. The example restores `true`. If you recorded `false`, restore `false`: pgflow then keeps the worker stopped, which preserves an intentionally disabled function.
110+
111+
7. ### Confirm the new worker runs
112+
113+
```sql
114+
SELECT worker_id, started_at, last_heartbeat_at
115+
FROM pgflow.workers
116+
WHERE function_name = 'your-worker-name'
117+
AND deprecated_at IS NULL
118+
AND last_heartbeat_at > now() - interval '6 seconds';
119+
```
120+
121+
Expect a row with a current heartbeat and a `started_at` after your deploy. The cron pings the re-enabled function within seconds and the new worker starts.
42122

43123
</Steps>
44124

45-
## Deployment Details
125+
## Why the Order Matters
126+
127+
Two mechanisms overlap during an update:
128+
129+
- **Worker lifetime.** An Edge Function keeps running for up to 150s (free tier) or 400s (paid) after deployment. Without deprecation, the old worker keeps processing tasks while the new version starts, so both versions run simultaneously.
130+
- **Automatic restart.** The `ensure_workers()` cron runs every second and pings every function whose `worker_functions` row is `enabled` and has no alive worker. Deprecated workers do not count as alive. If you deprecate workers while the row stays enabled, the cron pings the function within a second and starts another worker with the old, still-deployed code. The deprecation undoes itself.
131+
132+
This is why the sequence disables the function row first: deprecation then drains the workers, deployment swaps the code, and re-enabling starts exactly one worker with the new code.
133+
134+
## When the Drain or Restart Does Not Complete
135+
136+
### A deprecated worker keeps a current heartbeat
137+
138+
By default a deprecated worker stops heartbeating within about 11 seconds: 5s deprecation check plus the 6s freshness window. If the heartbeat stays current, list the workers of the function:
139+
140+
```sql
141+
SELECT worker_id, started_at, deprecated_at, last_heartbeat_at
142+
FROM pgflow.workers
143+
WHERE function_name = 'your-worker-name'
144+
ORDER BY started_at DESC;
145+
```
146+
147+
A row with `deprecated_at IS NULL` and a `started_at` after your deprecation means something started the function again. Two sources do this: the cron (the function row was re-enabled) and any direct HTTP request to the function (this starts a worker even while the row is disabled, because the function is still deployed). Confirm `enabled = false`, find the source of the restart, then repeat step 3 to deprecate the new worker.
46148

47-
### Why Worker Deprecation
149+
### Tasks stay in started
48150

49-
Without deprecation, old workers run for up to 150s (free tier) or 400s (paid) after deployment, causing both versions to process tasks simultaneously.
151+
A task that keeps matching the in-flight query means the deprecated worker stopped before the task finished. Inspect it:
152+
153+
```sql
154+
SELECT run_id, step_slug, task_index, requeued_count, last_requeued_at, permanently_stalled_at
155+
FROM pgflow.step_tasks
156+
WHERE status = 'started'
157+
AND last_worker_id IN (
158+
SELECT worker_id
159+
FROM pgflow.workers
160+
WHERE function_name = 'your-worker-name'
161+
AND deprecated_at IS NOT NULL
162+
);
163+
```
50164

51-
Deprecation fixes this: workers check their status every 5 seconds and immediately stop polling for new tasks when deprecated, while finishing current work.
165+
The stalled-task cron requeues such tasks after the step timeout plus 30 seconds, up to 3 attempts, then marks them with `permanently_stalled_at`. Requeued tasks return to `queued` and the next worker picks them up. Check the step handler logs for the root cause. See [Troubleshooting Stalled Tasks](/deploy/troubleshooting-stalled-tasks/).
52166

53-
### Monitoring Workers
167+
### No new worker after re-enabling
54168

55-
Track deployment progress with this query:
169+
Check whether the cron is pinging the function:
170+
171+
```sql
172+
SELECT function_name, enabled, last_invoked_at
173+
FROM pgflow.worker_functions
174+
WHERE function_name = 'your-worker-name';
175+
```
176+
177+
- `enabled` is `false`: step 6 did not restore the row. Restore it.
178+
- `last_invoked_at` does not advance: the cron is not invoking the function. Check the cron job exists and the vault secrets are set, because the cron builds the function URL and authorizes the ping from them:
179+
180+
```sql
181+
SELECT jobname, schedule, command
182+
FROM cron.job
183+
WHERE jobname = 'pgflow_ensure_workers';
184+
```
185+
186+
See [Configure Secrets](/deploy/supabase/configure-secrets/).
187+
188+
- `last_invoked_at` advances but no worker row appears: the ping reaches Supabase but the function fails to start. Check the Edge Function logs in the Supabase dashboard. See [Monitor workers health](/deploy/monitor-workers-health/).
189+
190+
## Monitoring Workers
191+
192+
Track all functions with this query:
56193

57194
```sql
58195
SELECT
@@ -63,4 +200,3 @@ FROM pgflow.workers
63200
WHERE last_heartbeat_at > now() - interval '6 seconds'
64201
GROUP BY function_name;
65202
```
66-

‎pkgs/website/src/content/docs/deploy/worker-management.mdx‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,13 @@ SET deprecated_at = now()
3939
WHERE function_name = 'my-worker';
4040
```
4141

42+
<Aside type="caution" title="Updating a deployed worker?">
43+
Deprecation alone does not keep a worker stopped. While the function's row in
44+
`pgflow.worker_functions` stays `enabled`, the `ensure_workers()` cron restarts the
45+
function, and the restarted worker runs the currently deployed code. Follow
46+
[Update Deployed Flows](/deploy/supabase/update-deployed-flows/) for the safe sequence.
47+
</Aside>
48+
4249
```d2
4350
...@../../../assets/pgflow-theme.d2
4451

0 commit comments

Comments
 (0)