Kubernetes deployment

SINDyffuse runs locally via scripts/*.py and on Kubernetes via Job manifests in deploy/jobs/. Each job wraps the same Python entry points with cluster resources (CPU, GPU, PVC).

Layout

deploy/
  pvc.yaml                    # cluster PVC (apply once)
  scripts/
    preprocess-dataset-orchestrate.sh  # entry: full | ik | path-fit | moco
    k8s-orchestrate-lib.sh             # shared kubectl apply + wait helpers
    path-fit-orchestrate.sh            # single path-fit Job (sample → convert → fit)
    moco-track-orchestrate.sh          # moco workers → normalization (moco stage)
  components/
    cluster-config/           # edit image + PVC here (applies to all jobs/dev pod)
  jobs/
    preprocess-dataset/
      base/                     # shared ConfigMap: PREPROCESS_NUM_SHARDS, SKIP_EXISTING
      inverse_kinematics/       # IndexedJob (180 × 4 CPU)
      fit-function-paths/       # single Job (MTP-welded path fit; skip if XML exists)
      moco-track/
        job.yaml                # IndexedJob MocoTrack workers (emptyDir scratch)
      normalization/            # Mean.npy / Std.npy after moco workers
    benchmark-moco-parallel/
    train-sindy/
    train-surrogate/
    train-diffusion/
      none/
      nimble/
      sindy/
  dev/                        # long-running interactive pod

Local	Kubernetes
`python scripts/preprocess_ik.py ...` then `preprocess_moco.py`	`./deploy/scripts/preprocess-dataset-orchestrate.sh [full\|ik\|path-fit\|moco]`
`python scripts/compute_normalization.py ...`	`./deploy/scripts/preprocess-dataset-orchestrate.sh moco` or `kubectl apply -k deploy/jobs/preprocess-dataset/normalization`
`python scripts/fit_rajagopal_function_paths.py ...` (Mode A local)	`./deploy/scripts/preprocess-dataset-orchestrate.sh path-fit YOUR_NAMESPACE`
`python scripts/benchmark_moco_parallel.py ...`	`kubectl apply -k deploy/jobs/benchmark-moco-parallel`
`python scripts/train_sindy.py ...`	`kubectl apply -k deploy/jobs/train-sindy`
`python scripts/train_surrogate.py ...`	`kubectl apply -k deploy/jobs/train-surrogate`
`python scripts/train_diffusion.py ...`	`kubectl apply -k deploy/jobs/train-diffusion/{none,nimble,sindy}`

Container image (GHCR)

The SINDyffuse runtime image is built from env/Dockerfile (CUDA + conda env sindyffuse from env/environment.yaml).

Published image

GitHub Actions publishes to GitHub Container Registry when env/Dockerfile or env/environment.yaml changes on main, or when you push a version tag (v*):

ghcr.io/rose-stl-lab/sindyffuse:latest
ghcr.io/rose-stl-lab/sindyffuse:sha-<commit>
ghcr.io/rose-stl-lab/sindyffuse:v1.0.0   # on git tag v1.0.0

Pull (after the package is public, or with read:packages auth):

docker pull ghcr.io/rose-stl-lab/sindyffuse:latest

Trigger a manual build: Actions → Docker image → Run workflow.

After the first publish, open the package on GitHub (Packages → sindyffuse → Package settings) and set visibility to Public so clusters can pull without credentials.

GitHub limits (free accounts)

Limit	Applies to SINDyffuse image?
500 MB GitHub Packages storage (npm, etc.)	No — GHCR containers are billed separately
GHCR storage/bandwidth	Currently free (GitHub may change with 1-month notice)
10 GB max per image layer	Yes — conda + PyTorch layers must stay under this (typical builds do)
10 minute upload timeout per layer	Yes — first push can be slow; rebuilds use GHA cache

Expect a large image (~8–15 GB total, CUDA + PyTorch + OpenSim + Nimble). That is normal for this stack and is fine on GHCR today.

CI build failed with No space left on device? GitHub's default runners have limited disk; the workflow runs a free-disk-space step before building. Re-run Actions → Docker image after pulling the latest workflow fix.

Build locally

From the repo root:

docker build -f env/Dockerfile -t ghcr.io/rose-stl-lab/sindyffuse:latest .
docker push ghcr.io/rose-stl-lab/sindyffuse:latest   # requires GHCR login

Nautilus (NRP) compliance

Apply jobs to your namespace (not stored in repo manifests):

kubectl apply -k deploy/jobs/preprocess-dataset/inverse_kinematics -n YOUR_NAMESPACE

Checklist aligned with NRP cluster policies:

Rule	Our setup
limits within 20% of requests	All jobs use limits = requests (Guaranteed QoS)
> ~100 pods: limit = request	Moco (180 shards): 20 CPU / 16Gi limits = requests ✓
1 CPU + 2Gi exemption	Small pods only; IK workers are 180 × 4 CPU / 4Gi (Guaranteed) ✓
Batch jobs, not sleep infinity	Jobs run Python scripts to completion ✓
No GPUs on CPU-only preprocess	Preprocess jobs request CPU/memory only ✓
PVC	`rook-cephfs`, `ReadWriteMany` (standard Nautilus CephFS) ✓
Large parallel submits	180 moco pods is a large footprint; coordinate with namespace admins if scheduling is slow

backoffLimit: Indexed preprocess jobs retry failed shard pods a limited number of times before the Job fails (IK/moco: 64). Single-pod jobs (normalization, path-fit): 3.

SKIP_EXISTING: Default on via shared ConfigMap sindyffuse-preprocess-config (SKIP_EXISTING=1, PREPROCESS_NUM_SHARDS=180 in deploy/jobs/preprocess-dataset/base/). Omit or set SKIP_EXISTING=0 in an overlay to force full reprocess.

Image: deploy/components/cluster-config rewrites sindyffuse:latest → ncking/sindyffuse:latest. Every job kustomization must include that component.

Edit deploy/components/cluster-config/ for registry tag and PVC name for your environment.

Quick start

1. Use the published image (or build locally)

Default image (from GHCR, after CI has run once):

ghcr.io/rose-stl-lab/sindyffuse:latest

Or build yourself:

docker build -f env/Dockerfile -t ghcr.io/rose-stl-lab/sindyffuse:latest .
docker push ghcr.io/rose-stl-lab/sindyffuse:latest   # optional

See Container image (GHCR) for limits and visibility settings.

2. Configure cluster settings

Edit deploy/components/cluster-config/ (single template for every manifest):

File	Set
`kustomization.yaml`	`images.newName`, `images.newTag`
`pvc-patch.yaml`, `pvc-patch-pod.yaml`	`claimName` (PVC name)

To keep local overrides out of git, copy the folder and point a job’s kustomization.yaml at your copy:

cp -r deploy/components/cluster-config deploy/overlays/local/cluster-config
# components: ../overlays/local/cluster-config

3. Create storage (if needed)

Apply once per cluster:

kubectl apply -f deploy/pvc.yaml

The PVC should contain:

Repo checkout at /mnt/SINDyffuse
HumanML3D at /mnt/SINDyffuse/datasets/HumanML3D

4. Apply a job

# Full preprocess pipeline (local scripts apply + wait on cluster Jobs)
./deploy/scripts/preprocess-dataset-orchestrate.sh full YOUR_NAMESPACE

# Or individual stages:
./deploy/scripts/preprocess-dataset-orchestrate.sh ik YOUR_NAMESPACE
./deploy/scripts/preprocess-dataset-orchestrate.sh path-fit YOUR_NAMESPACE
./deploy/scripts/preprocess-dataset-orchestrate.sh moco YOUR_NAMESPACE

# Or apply a single work Job directly:
kubectl apply -k deploy/jobs/preprocess-dataset/inverse_kinematics -n YOUR_NAMESPACE

# Train SINDy (2× GPU)
kubectl apply -k deploy/jobs/train-sindy

# Train activation surrogate (2× GPU; auto torchrun in Python; requires Mean.npy/Std.npy)
kubectl apply -k deploy/jobs/train-surrogate

# Train diffusion — pick guidance mode
kubectl apply -k deploy/jobs/train-diffusion/none
kubectl apply -k deploy/jobs/train-diffusion/nimble
kubectl apply -k deploy/jobs/train-diffusion/sindy

Preview a rendered preprocess manifest:

kubectl kustomize deploy/jobs/preprocess-dataset/inverse_kinematics

Delete jobs before a manual re-run:

kubectl delete job sindyffuse-preprocess-ik \
  sindyffuse-fit-function-paths \
  sindyffuse-preprocess-moco-track sindyffuse-compute-normalization --ignore-not-found -n YOUR_NAMESPACE

Dev pod

Long-running pod for interactive work on the cluster (same image + PVC as jobs):

kubectl apply -k deploy/dev
kubectl get pod sindyffuse-dev -w
kubectl exec -it sindyffuse-dev -- bash -l

Inside the shell, conda activate sindyffuse is already configured. Run scripts the same way as locally:

python scripts/preprocess_ik.py --max_motions 1
python scripts/preprocess_moco.py --max_motions 1

Delete when done:

kubectl delete pod sindyffuse-dev

Default resources: 8–32 CPU, 32–64Gi memory. Edit deploy/dev/pod.yaml to add GPUs or change limits.

In-container layout

All manifests assume the PVC is mounted at /mnt with the repo at /mnt/SINDyffuse (workingDir on every pod/job). HumanML3D lives at /mnt/SINDyffuse/datasets/HumanML3D.

Only preprocess jobs accept optional env: PREPROCESS_NUM_SHARDS, SKIP_EXISTING (shared ConfigMap), PATH_FIT_SAMPLE_MOTIONS, PATH_FIT_NUM_THREADS, PATH_FIT_NUM_WORKERS, PATH_FIT_FORCE, MAX_MOTIONS, MOCO_TEMP_DIR / TMPDIR (moco scratch).

Preprocess uses Indexed worker Jobs (parallelism=completions=180 for inverse kinematics and moco). Motions are length-balanced across shards (file-size proxy). Path-fit is a single Job and is skipped when the FunctionBasedPathSet XML already exists. Pipeline sequencing is done by local scripts in deploy/scripts/ (kubectl apply + kubectl wait from your laptop — no in-cluster orchestrator Jobs or RBAC). Normalization merges moco shard manifests after moco workers finish.

Resource profiles

Job	CPUs	Memory	GPUs
dev	8–32	32–64Gi	— (add in pod.yaml if needed)
preprocess-dataset/inverse_kinematics	180 × 4	180 × 4Gi	—
preprocess-dataset/moco-track	180 × 20	180 × 16Gi (+ emptyDir `/scratch`)	—
preprocess-dataset/normalization	1	2Gi	—
preprocess-dataset/fit-function-paths	128	128Gi	—
benchmark-moco-parallel	64	64Gi	—
train-sindy	16	64Gi	1
train-surrogate	16	64Gi	1
train-diffusion/*	16	64Gi	1

Edit deploy/jobs/<name>/job.yaml to match your nodes.

Multi-GPU training

Training jobs source deploy/scripts/job-env.sh (conda + runtime env) and invoke the same Python entry points used locally, e.g. python scripts/train_diffusion.py --guidance sindy --preload. Run directories, resolved configs, preflight checks, and latest symlinks are handled inside the scripts (common/run_setup.py).

Each trainer calls maybe_relaunch_with_torchrun() in common/distributed.py, which:

Reads NPROC_PER_NODE (set in job yaml to match nvidia.com/gpu, default 1)
Else counts CUDA_VISIBLE_DEVICES
Else uses torch.cuda.device_count()
Caps the process count by the number of CUDA devices that can actually run kernels
Re-execs under torchrun when count > 1 and distributed.enabled is not false

If the pod only exposes one working GPU, training automatically falls back to single-GPU mode instead of launching a broken 2-process job.

Set NPROC_PER_NODE=1 or SINDYFFUSE_NO_TORCHRUN=1 to force single-process training. Startup logs include [distributed/gpu] with rank, world size, device count, and /dev/nvidia* nodes.

Path fit (Job 2)

Mode A (local): python scripts/fit_rajagopal_function_paths.py --sample_motions 200 — skips if XML exists; optional --force, --num_workers / --num_threads.

Mode C (cluster): single Job (same command as local). Orchestrator skips when XML is present unless PATH_FIT_FORCE=1:

./deploy/scripts/preprocess-dataset-orchestrate.sh path-fit YOUR_NAMESPACE
# or: kubectl apply -k deploy/jobs/preprocess-dataset/fit-function-paths -n YOUR_NAMESPACE

Job	Resources	Role
`sindyffuse-fit-function-paths`	128 CPU, 128Gi	sample → B3D→`.mot` (32 workers) → merge → OpenSim path fit (128 threads) on MTP-welded Rajagopal

Run after IK (Job 1) and before MocoTrack (Job 3).

Pipeline order

./deploy/scripts/preprocess-dataset-orchestrate.sh ik (or apply inverse_kinematics/ Job)
./deploy/scripts/preprocess-dataset-orchestrate.sh path-fit
./deploy/scripts/preprocess-dataset-orchestrate.sh moco
train-sindy
train-surrogate
train-diffusion/{none,nimble,sindy}

Single entry point for the full preprocess pipeline:

./deploy/scripts/preprocess-dataset-orchestrate.sh full YOUR_NAMESPACE

Results layout

Each trainer writes timestamped runs under results/<family>/runs/ and updates results/<family>/latest when training completes (see common/run_setup.py).

Local development

No Kubernetes required for development:

conda env create -f env/environment.yaml
conda activate sindyffuse
python scripts/preprocess_ik.py --max_motions 1
python scripts/preprocess_moco.py --max_motions 1

See the root README.md for the full local pipeline.

Navigation Menu

Uh oh!

Kubernetes deployment

Layout

Container image (GHCR)

Published image

GitHub limits (free accounts)

Build locally

Nautilus (NRP) compliance

Quick start

1. Use the published image (or build locally)

2. Configure cluster settings

3. Create storage (if needed)

4. Apply a job

Dev pod

In-container layout

Resource profiles

Multi-GPU training

Path fit (Job 2)

Pipeline order

Results layout

Local development

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Kubernetes deployment

Layout

Container image (GHCR)

Published image

GitHub limits (free accounts)

Build locally

Nautilus (NRP) compliance

Quick start

1. Use the published image (or build locally)

2. Configure cluster settings

3. Create storage (if needed)

4. Apply a job

Dev pod

In-container layout

Resource profiles

Multi-GPU training

Path fit (Job 2)

Pipeline order

Results layout

Local development