Payload 4.0 stable timeline, and backport of the atomic job claim (#17441) to 3.x?

[tiendungnguyen24]

We run Payload 3.x jobs with more than one runner on the same queue and hit the non-atomic job claim: updateJobs in @payloadcms/db-mongodb claims a job in three steps (find → updateMany → find), so two runners can claim and run the same job. For now we limit ourselves to a single runner.

#17441 (feat!: support multiple workers on the same queue) fixes this, but it is only in 4.0, which has canary builds only.

1. Is there a rough timeline for a stable 4.0?
2. Since v3 is in maintenance mode, is a backport of the atomic claim to 3.x possible, at least findOneAndUpdate for limit: 1? It fixes a correctness bug (duplicate job runs), not a new feature.
3. Until then, is a single runner per queue the recommended setup on 3.x?

Thanks!

[tisankanj]

For the interim v3 setup, I would keep one active runner per queue and prevent overlapping run calls. That is a mitigation for this claim race, not an exactly-once guarantee or an official release recommendation. A single process can still overlap two cron/manual/API-triggered runs, and a rolling deployment can temporarily have both old and new runner replicas alive.

Your source-level diagnosis matches the MongoDB adapter in v3.90.2. In the limited, non-ID path, it finds candidate IDs, replaces the query with an _id filter, performs updateMany, and then reads those IDs back. That last read does not establish which runner won the claim. Setting limit: 1 alone still follows that path; it does not switch to findOneAndUpdate.

PR #17441 is useful here because its MongoDB fix still uses find/update/find. The difference is ownership: a conditional claim writes a unique processingToken, and the final read returns only jobs claimed with that runner's token. So three operations by themselves are not the issue; preserving eligibility during the update and identifying the winner are the important parts. The PR also explicitly separates claim exclusivity from leases/crash recovery.

Until you have a released fix validated for your version, I would check:

• Only one consumer is active for this queue across all replicas, including deployment overlap.
• autoRun, an external cron, a CLI runner and manual/API calls are not independently polling the same queue. Wait for one run to finish before starting the next.
• Job side effects tolerate retries, using a stable business-operation ID. For an external API that supports idempotency keys, reuse that key when retrying. A local “done” flag alone cannot protect the window where the external action succeeds and the worker crashes before recording completion.

A findOneAndUpdate branch for limit: 1 could address the MongoDB claim operation, but it needs to preserve the eligibility filter, sort, session and adapter transformations, and return only the successfully claimed document. I would treat it as a reviewed adapter change with a two-runner regression test, rather than patching just the final read or reducing the limit.

I can't give a 4.0 stable date or confirm a v3 backport commitment; those need a maintainer response. I reviewed the released source and the merged PR, but have not run your deployment or a MongoDB concurrency reproduction.

[kevinjosh10]

Hi @community! For multi-runner queues hitting the non-atomic updateJobs claim in @payloadcms/db-mongodb:

---

Workaround Until Backported

PR #17441 switches the MongoDB claim from three steps (find → updateMany → find) to an atomic findOneAndUpdate({ status: 'pending' }, { $set: { status: 'running', lockedAt: new Date() } }).

While waiting for the 3.x patch release:

• Partition worker queues by queue name (e.g. queue-worker-1, queue-worker-2).
• Or run the queue workers with a single runner concurrency lock (concurrency: 1) to eliminate the race condition.

[Sisuthros]

Short version for your three questions. It is based on reading this thread, #17441, #18565 and #18709. I haven't reproduced it myself, so treat it as a code-reading diagnosis.

Timeline / backport. I can't give you a 4.0 date, and I don't see a maintainer commitment to a 3.x release in the threads. The work is actually happening in #18565 (the 3.x backport request) and in the community PR #18709 targeting 3.x. That PR currently has changes requested, so it is not a drop-in yet. The review found three problems: Mongo batch claims can leave jobs marked processing but excluded from the read-back (so they never run), concurrent runByID is still unguarded, and the hook/depth paths in queues/utilities/updateJob.ts bypass the adapter change. A comment with your adapter and version on #18565 is probably the fastest way to get a maintainer response.

On how #17441 works (the two earlier answers here disagree). It is not a single findOneAndUpdate. On Mongo it keeps find / update / find, but the update is conditional and stamps a unique processingToken, so each runner's final read returns only the jobs it actually won. Drizzle uses one conditional UPDATE ... RETURNING. So a naive findOneAndUpdate patch for limit: 1 covers single-job claims but not batches, and the real fix needs the new processingToken field.

Is one runner per queue enough until then? It shrinks the window but does not close it. The race is not only between replicas: anything that triggers a run while another run is in flight can overlap. That includes autoRun plus an external cron hitting /api/payload-jobs/run, a manual or CLI run, and rolling deploys where old and new instances are both up. In #18565 the reporter hit the duplicate in roughly 1 of 3 attempts with two run calls about a second apart.

What I would do on 3.x until a fix ships. (a) Use exactly one trigger per queue (autoRun or an external cron, not both), and make that scheduler non-overlapping: skip if the previous run is still going. (b) Treat every task as at-least-once and key the side effect on the business operation, not the job id. For example, a scheduled publish checks "already published at this version" before writing. On Postgres, the advisory lock around the run endpoint described in #18565 is a workable stopgap. (c) With deleteJobOnComplete: true, the loser of a race throws the reading 'log' TypeError. Alert on it rather than swallowing it, because it tells you two runners overlapped. (d) On Mongo, claiming each candidate with its own findOneAndUpdate (the stopgap shared in #18565) avoids the batch-strand problem, at the cost of one round trip per job.

None of this gives exactly-once; it makes the duplicate harmless while the claim fix lands.

If you want an independent test of this boundary, we run a fixed-scope 100 EUR triage with a written PASS/FAIL finding: https://clawguard-teal.vercel.app/clawguard/landing.html

(Drafted with an AI assistant, Claude.)