feat: add per-node stage worker sizing

[praateekmahajan]

Summary

• Add backend-neutral ProcessingStage.num_workers_per_node() and with_(num_workers_per_node=...) configuration.
• Translate per-node sizing across Ray Data, Ray Actor Pool, and Xenna while keeping the legacy Xenna stage-spec key compatible.
• Size Ray Data pools as ceil(workers_per_node * alive_nodes) with best-effort SPREAD placement and head-node-aware counting.
• Route document download stages through the common policy and default Arxiv, Common Crawl, and Wikipedia downloads to two workers per node.
• Centralize conflicting worker-sizing validation and update current Fern/backend guidance.

API changes

• num_workers, num_workers_per_node, and Ray Data actor-pool min/max/initial sizing are mutually exclusive.
• ImageDuplicatesRemovalStage(num_workers_per_node=...) is replaced by ImageDuplicatesRemovalStage(...).with_(num_workers_per_node=...).
• Existing xenna_stage_spec()["num_workers_per_node"] configurations remain supported when the common worker hooks are unset.

Validation

• Repository pre-commit hooks pass.
• 88 targeted stage/backend/unit tests pass, plus the new document-download delegation path was exercised directly. The existing local environment does not include the optional text packages needed to collect the full download test module; CI will run it with repository extras.

Replaces #2198.
Fixes #2197.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-praateek-num-workers-per-node.docs.buildwithfern.com/nemo/curator

Here are the markdown pages you've updated:

• nemo/curator/api/reference/api-reference/executors/xenna-executor
• nemo/curator/api/reference/api-reference/processing-stage
• nemo/curator/reference/infra/execution-backends
• nemo/curator/reference/infra/stage-worker-sizing

[greptile-apps]

Greptile Summary

This PR adds backend-neutral per-node worker sizing and translates it into backend-specific worker-pool configuration.

• Adds ProcessingStage.num_workers_per_node() and corresponding with_() support.
• Applies per-node sizing across Ray Data, Ray Actor Pool, and Xenna.
• Centralizes worker-sizing conflict and value validation.
• Routes document download concurrency through the shared sizing policy.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
nemo_curator/backends/utils.py	Centralizes validation of common per-node worker counts, including rejection of NaN and infinity.
nemo_curator/stages/base.py	Adds the shared per-node sizing hook, immutable configuration override, and conflicting-sizing validation.
nemo_curator/backends/ray_data/adapter.py	Converts per-node sizing into fixed Ray Data pools using alive-node counts and best-effort spread placement.
nemo_curator/backends/ray_actor_pool/utils.py	Converts per-node sizing into a cluster-wide actor request and caps it against available resources.
nemo_curator/backends/xenna/executor.py	Maps the common sizing hook to Xenna while retaining the legacy stage-spec key.
nemo_curator/stages/text/download/base/download.py	Delegates document downloader concurrency through the common stage sizing policy.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Stage["ProcessingStage.num_workers_per_node()"] --> Validate["Validate finite positive value"]
    Validate --> RayData["Ray Data: ceil(value × alive nodes)"]
    Validate --> ActorPool["Ray Actor Pool: ceil(value × alive nodes), then cap"]
    Validate --> Xenna["Xenna: forward per-node value"]

Loading

_{Reviews (2): Last reviewed commit: "fix: reject non-finite per-node worker c..." | Re-trigger Greptile}

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[ayushdg]

Thoughts on SPREAD vs STRICT_SPREAD for this given that num_workers_per_node seems to be a fixed/opinionated config?

[ayushdg]

I think the docs mention this but we're not guaranteed to avoid scheduling on the head node in some cases.
Do you forsee this causing issues for setups?