[benchmarking] Add separate nemo-curator-benchmarking package

[rlratzel]

Motivation

Curator benchmarks require dependencies that should not be redistributed in the released Curator image. The previous approach relied on a separate benchmarking Dockerfile and duplicated environment setup logic outside the Curator repo, which made dependency updates easy to miss and required separate follow-up changes for internal benchmark automation.

This change introduces nemo-curator-benchmarking as a separate, manually-installed package. Benchmark authors can update benchmark scripts, configs, tools, and benchmark-only dependency declarations together in one Curator PR to provide a single source of truth for Curator benchmarks and their benchmark-owned dependencies.

The package also gives downstream automation, such as NeMo-CI, a reusable place to call shared benchmark utilities instead of duplicating logic for parsing configs, merging YAML files, and listing benchmark entries. It also moves Python dependencies that NeMo-CI currently installs directly at benchmark runtime into Curator-owned package metadata, while leaving Curator feature-stack dependencies in the main Curator package extras.

It also makes it easier to benchmark an older released Curator image using the latest benchmark suite instead of being limited to the scripts and configs baked into that older image.

Summary

• Add a separate nemo-curator-benchmarking package with a curator-benchmark console entrypoint.
• Add the curator_benchmarking Python package behind the CLI and future shared benchmarking utilities.
• Add benchmark-package dependency extras for sinks, audio benchmark-only helpers, and Nemotron-Parse benchmark runtime dependencies.
• Keep Curator feature-stack dependencies, such as audio, video, CUDA/vLLM, and interleaved data processing, owned by the main Curator package extras instead of duplicating them.
• Support curator-benchmark run/list/check/setup/shell in the current environment, a Docker image, or an existing running container.
• Move the existing benchmark runner into the benchmarking package while keeping python benchmarking/run.py ... as a compatibility shim.
• Replace benchmarking/tools/run.sh with a thin compatibility wrapper.
• Remove the dedicated benchmarking Dockerfile, Docker build helper, and bash variable generator.
• Update benchmark script path resolution so installed benchmark scripts and data-prep scripts work from the benchmark package.

New Capabilities

• Run benchmarks in the active environment with curator-benchmark run.
• Start a standard Curator image with curator-benchmark run --image ....
• Reuse a prepared running container with curator-benchmark run --container ....
• Install or skip benchmark package setup with --benchmark-setup auto|always|never.
• Check a benchmark environment with curator-benchmark check.
• Install benchmark-owned dependency sets with extras such as ./benchmarking[audio], ./benchmarking[nemotron_parse], or ./benchmarking[all].
• Reuse curator_benchmarking modules from downstream automation instead of reimplementing config/listing helpers.
• Keep direct benchmark script execution supported via python benchmarking/scripts/<script>.py ....
• Preserve legacy python benchmarking/run.py ... and benchmarking/tools/run.sh ... workflows as wrappers.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

The PR separates the benchmark runner and benchmark-owned dependencies into an installable nemo-curator-benchmarking package, adds a unified CLI for local and Docker targets, and retains compatibility wrappers.

• Adds run, list, check, setup, and shell command dispatch.
• Adds image and existing-container execution with benchmark package setup modes.
• Moves the existing runner behind the package while retaining legacy entrypoints.
• Defines benchmark-only dependency extras and updates path resolution and documentation.

Confidence Score: 3/5

The PR should not merge until existing-container config handling and automatic detection of missing requested extras are corrected.

Existing-container runs can be rejected by an inappropriate host-side path check, and automatic setup can skip required optional dependencies whenever the base package already imports.

Files Needing Attention: benchmarking/curator_benchmarking/targets/docker.py

Important Files Changed

Filename	Overview
benchmarking/curator_benchmarking/targets/docker.py	Adds Docker target construction, setup, path translation, and command execution; existing-container config validation and incomplete auto-setup detection can prevent supported runs.
benchmarking/curator_benchmarking/commands/run.py	Moves the existing benchmark runner into the package while preserving entry selection, setup ordering, result processing, and sink lifecycle behavior.
benchmarking/curator_benchmarking/cli.py	Adds top-level command and target dispatch while retaining legacy argument behavior.
benchmarking/pyproject.toml	Defines the separate benchmark package, console entrypoint, packaged modules, data files, and benchmark-owned dependency extras.
benchmarking/run.py	Replaces the previous runner implementation with a compatibility shim to the new CLI.
benchmarking/runner/entry.py	Resolves benchmark scripts from the package while allowing the Curator-under-test checkout to be selected separately.
tests/benchmarking/test_curator_benchmarking_cli.py	Covers basic CLI dispatch and image-target config rewriting but does not cover existing-container paths or incomplete extras in auto setup.

Sequence Diagram

sequenceDiagram
    participant User
    participant CLI as curator-benchmark CLI
    participant Launcher as Docker target launcher
    participant Target as Image / existing container
    participant Runner as Benchmark runner
    User->>CLI: run/list/check/setup/shell
    CLI->>Launcher: Parse target and setup options
    alt Image target
        Launcher->>Target: docker run with suite, configs, and data mounts
    else Existing container
        Launcher->>Target: docker exec using prepared mounts
    else Current environment
        CLI->>Runner: Dispatch command directly
    end
    Launcher->>Target: Check or install benchmark package extras
    Target->>Runner: Execute curator-benchmark
    Runner-->>User: Results and exit status

Loading

_{Reviews (1): Last reviewed commit: "feat(benchmarking): add nemo-curator-ben..." | Re-trigger Greptile}

[greptile-apps]

Container config host validation

When --container receives a config path that exists inside the prepared container but not at the same host path, _containerize_command_args validates it through _config_file_mount_pairs before invoking docker exec, causing the documented existing-container workflow to exit with Config file does not exist.

[greptile-apps]

Auto setup misses extras

When the base benchmark package imports but a requested extra such as sinks is absent, auto mode treats setup as complete and skips the extras installation, causing the run to fail later while importing or initializing the corresponding benchmark helper.