You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 99bef82
Browse filesBrowse the repository at this point in the historyBrowse files
docs: fence worker_functions row before deprecating workers in update guide (#670)
The production Update Deployed Flows guide told operators to deprecate
old workers, deploy, and wait for cron. While the pgflow.worker_functions
row stays enabled, the ensure_workers() cron (1s interval) can start
another instance of the old deployed function between deprecation and
deployment, because deprecated workers do not count as alive for the
restart check.
Rewrite the guide around a disable fence for one worker function:
- record the current enabled value, then set enabled = false
- deprecate its live pgflow.workers rows
- verify polling stopped (heartbeat window) and in-flight work drained
(step_tasks.status leaves 'started' for the deprecated workers)
- deploy the replacement while the row stays disabled
- restore the exact recorded enabled value, not an inferred one
- confirm the replacement worker reports a current heartbeat
Add failure guidance covering a worker that does not drain or does not
restart, with inspection SQL (worker listing, stuck task detail,
last_invoked_at, cron.job entry). Link the corrected sequence from the
Worker Deprecation section of the Worker Management guide.
Per-step, alias-version, and shared-queue rollout instructions stay with
issues #651, #648, and #652; deliberately not documented here.
Closes#654
Co-authored-by: Agent <agent@pgflow.dev>
When updating flows that are already deployed to production, use the deprecation workflow to prevent old and new worker versions from processing tasks simultaneously.
13
+
When updating flows that are already deployed to production, follow this sequence to prevent old and new worker versions from processing tasks simultaneously. The order matters: the `ensure_workers()` cron restarts any function whose `worker_functions` row is `enabled`, so you must disable the row before you deprecate workers.
14
+
15
+
See [Worker Management](/deploy/worker-management/) for how worker registration, deprecation, and heartbeats work.
14
16
15
17
<Asidetype="note"title="First Time Deployment?">
16
18
If you haven't deployed to production yet, see [Deploy Your First Flow](/deploy/supabase/deploy-first-flow/) for initial setup instructions.
17
19
</Aside>
18
20
19
21
## Update Workflow
20
22
23
+
Throughout, replace `your-worker-name` with your Edge Function name.
24
+
21
25
<Steps>
22
26
23
-
1.**Deprecate old workers**
27
+
1.### Record the current enabled state
28
+
29
+
```sql
30
+
SELECT function_name, enabled
31
+
FROMpgflow.worker_functions
32
+
WHERE function_name ='your-worker-name';
33
+
```
34
+
35
+
Note the `enabled` value. You restore exactly this value in step 6, even if it is `false`.
36
+
37
+
2.### Disable automatic restarts
38
+
39
+
```sql
40
+
UPDATEpgflow.worker_functions
41
+
SET enabled = false
42
+
WHERE function_name ='your-worker-name';
43
+
```
44
+
45
+
While `enabled` is `false`, the cron stops pinging this function. Disable the row before deprecating workers, otherwise the cron restarts the old worker within a second of the deprecation.
46
+
47
+
3.### Deprecate live workers
48
+
24
49
```sql
25
50
UPDATEpgflow.workers
26
-
SET deprecated_at =NOW()
51
+
SET deprecated_at =now()
27
52
WHERE function_name ='your-worker-name'
28
53
AND deprecated_at IS NULL;
29
54
```
30
-
This stops old workers from polling for new tasks while they finish current work.
31
55
32
-
2.**Deploy new version**
56
+
Workers check their deprecation status every 5 seconds. A deprecated worker stops polling for new tasks and finishes the tasks it already claimed.
57
+
58
+
4.### Wait for the drain to finish
59
+
60
+
A worker stopped polling when none of its function's workers has a current heartbeat:
AND last_heartbeat_at > now() - interval '6 seconds';
67
+
```
68
+
69
+
This returns no rows once every worker of the function stopped polling.
70
+
71
+
In-flight work finished when no task claimed by a deprecated worker is still running:
72
+
73
+
```sql
74
+
SELECT run_id, step_slug, task_index, started_at
75
+
FROMpgflow.step_tasks
76
+
WHERE status ='started'
77
+
AND last_worker_id IN (
78
+
SELECT worker_id
79
+
FROMpgflow.workers
80
+
WHERE function_name ='your-worker-name'
81
+
AND deprecated_at IS NOT NULL
82
+
);
83
+
```
84
+
85
+
This returns no rows when every claimed task reached a terminal state (`completed` or `failed`) or was requeued back to `queued`.
86
+
87
+
<Asidetype="note">
88
+
pgflow does not record completion callbacks as separate rows. The observable signal is the task row leaving `started`: the worker writes the final task state only after the handler and its completion callbacks finish.
89
+
</Aside>
90
+
91
+
Repeat both queries until they return no rows. The drain takes as long as the longest in-flight task. If the function had no live workers, both queries return no rows immediately.
92
+
93
+
5.### Deploy the new version
94
+
33
95
```bash frame="none"
34
96
npx supabase functions deploy your-worker-name
35
97
```
36
98
37
-
3.**Wait for new worker to start**
99
+
The function row stays disabled during deployment, so nothing restarts the worker while the new code deploys.
100
+
101
+
6.### Restore the recorded enabled state
38
102
39
-
The pgflow cron automatically starts the new worker within seconds.
103
+
```sql "true"
104
+
UPDATEpgflow.worker_functions
105
+
SET enabled = true
106
+
WHERE function_name ='your-worker-name';
107
+
```
40
108
41
-
See [Worker Management](/deploy/worker-management/) for details.
109
+
Set the value you recorded in step 1. The example restores `true`. If you recorded `false`, restore `false`: pgflow then keeps the worker stopped, which preserves an intentionally disabled function.
110
+
111
+
7.### Confirm the new worker runs
112
+
113
+
```sql
114
+
SELECT worker_id, started_at, last_heartbeat_at
115
+
FROMpgflow.workers
116
+
WHERE function_name ='your-worker-name'
117
+
AND deprecated_at IS NULL
118
+
AND last_heartbeat_at > now() - interval '6 seconds';
119
+
```
120
+
121
+
Expect a row with a current heartbeat and a `started_at` after your deploy. The cron pings the re-enabled function within seconds and the new worker starts.
42
122
43
123
</Steps>
44
124
45
-
## Deployment Details
125
+
## Why the Order Matters
126
+
127
+
Two mechanisms overlap during an update:
128
+
129
+
-**Worker lifetime.** An Edge Function keeps running for up to 150s (free tier) or 400s (paid) after deployment. Without deprecation, the old worker keeps processing tasks while the new version starts, so both versions run simultaneously.
130
+
-**Automatic restart.** The `ensure_workers()` cron runs every second and pings every function whose `worker_functions` row is `enabled` and has no alive worker. Deprecated workers do not count as alive. If you deprecate workers while the row stays enabled, the cron pings the function within a second and starts another worker with the old, still-deployed code. The deprecation undoes itself.
131
+
132
+
This is why the sequence disables the function row first: deprecation then drains the workers, deployment swaps the code, and re-enabling starts exactly one worker with the new code.
133
+
134
+
## When the Drain or Restart Does Not Complete
135
+
136
+
### A deprecated worker keeps a current heartbeat
137
+
138
+
By default a deprecated worker stops heartbeating within about 11 seconds: 5s deprecation check plus the 6s freshness window. If the heartbeat stays current, list the workers of the function:
A row with `deprecated_at IS NULL` and a `started_at` after your deprecation means something started the function again. Two sources do this: the cron (the function row was re-enabled) and any direct HTTP request to the function (this starts a worker even while the row is disabled, because the function is still deployed). Confirm `enabled = false`, find the source of the restart, then repeat step 3 to deprecate the new worker.
46
148
47
-
### Why Worker Deprecation
149
+
### Tasks stay in started
48
150
49
-
Without deprecation, old workers run for up to 150s (free tier) or 400s (paid) after deployment, causing both versions to process tasks simultaneously.
151
+
A task that keeps matching the in-flight query means the deprecated worker stopped before the task finished. Inspect it:
Deprecation fixes this: workers check their status every 5 seconds and immediately stop polling for new tasks when deprecated, while finishing current work.
165
+
The stalled-task cron requeues such tasks after the step timeout plus 30 seconds, up to 3 attempts, then marks them with `permanently_stalled_at`. Requeued tasks return to `queued` and the next worker picks them up. Check the step handler logs for the root cause. See [Troubleshooting Stalled Tasks](/deploy/troubleshooting-stalled-tasks/).
52
166
53
-
### Monitoring Workers
167
+
### No new worker after re-enabling
54
168
55
-
Track deployment progress with this query:
169
+
Check whether the cron is pinging the function:
170
+
171
+
```sql
172
+
SELECT function_name, enabled, last_invoked_at
173
+
FROMpgflow.worker_functions
174
+
WHERE function_name ='your-worker-name';
175
+
```
176
+
177
+
-`enabled` is `false`: step 6 did not restore the row. Restore it.
178
+
-`last_invoked_at` does not advance: the cron is not invoking the function. Check the cron job exists and the vault secrets are set, because the cron builds the function URL and authorizes the ping from them:
179
+
180
+
```sql
181
+
SELECT jobname, schedule, command
182
+
FROMcron.job
183
+
WHERE jobname ='pgflow_ensure_workers';
184
+
```
185
+
186
+
See [Configure Secrets](/deploy/supabase/configure-secrets/).
187
+
188
+
-`last_invoked_at` advances but no worker row appears: the ping reaches Supabase but the function fails to start. Check the Edge Function logs in the Supabase dashboard. See [Monitor workers health](/deploy/monitor-workers-health/).
189
+
190
+
## Monitoring Workers
191
+
192
+
Track all functions with this query:
56
193
57
194
```sql
58
195
SELECT
@@ -63,4 +200,3 @@ FROM pgflow.workers
63
200
WHERE last_heartbeat_at > now() - interval '6 seconds'
0 commit comments