Skip to content

Commit a8992b6

Browse files
wayrisewayrise
authored
Merge pull request #25 from OpenWAM-Official/docker/open-source-support
Add Docker workflows for GPU training, serving and offline deployment
2 parents d8dd33d + 3358295 commit a8992b6

28 files changed

Lines changed: 3340 additions & 25 deletions

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ jobs:
2121
run: pip install ruff
2222

2323
- name: Lint
24-
run: ruff check openwam/ scripts/ tests/
24+
run: ruff check openwam/ scripts/ tests/ docker/
2525

2626
tests:
2727
runs-on: ubuntu-latest

‎.github/workflows/docker.yml‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
name: Docker build and tests
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches: [main, 'docker/**']
7+
tags: ['v*']
8+
workflow_dispatch:
9+
10+
permissions:
11+
contents: read
12+
13+
concurrency:
14+
group: docker-${{ github.workflow }}-${{ github.ref }}
15+
cancel-in-progress: true
16+
17+
jobs:
18+
build-and-test:
19+
runs-on: ubuntu-24.04
20+
timeout-minutes: 120
21+
env:
22+
OPENWAM_IMAGE: openwam:ci-${{ github.sha }}
23+
steps:
24+
- uses: actions/checkout@v4
25+
- name: Validate configuration
26+
run: |
27+
docker compose --profile train config --quiet
28+
docker compose -f compose.yaml -f docker/compose.host.yaml config --quiet
29+
docker compose -f compose.yaml -f docker/compose.dev.yaml --profile dev --profile train config --quiet
30+
OPENWAM_WORKSPACE_DIR=/tmp/openwam-workspace OPENWAM_SOURCE_DIR=/tmp/openwam-workspace/feature docker compose -f compose.yaml -f docker/compose.dev.yaml -f docker/compose.worktree.yaml --profile dev --profile train config --quiet
31+
bash -n docker/entrypoint.sh docker/pip_install.sh scripts/train.sh
32+
# Only on this disposable GitHub-hosted runner. CUDA build + image load
33+
# need substantially more disk than a small Python application.
34+
- name: Make room for CUDA layers
35+
run: |
36+
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /usr/local/.ghcup /opt/hostedtoolcache
37+
df -h /
38+
available_kb=$(df --output=avail / | tail -1)
39+
if [ "$available_kb" -lt 60000000 ]; then
40+
echo 'CUDA CI needs at least 60 GB free; use a larger runner if this image layout changes.' >&2
41+
exit 1
42+
fi
43+
- uses: docker/setup-buildx-action@v3
44+
- name: Build and load the actual CUDA image
45+
uses: docker/build-push-action@v7
46+
with:
47+
context: .
48+
file: docker/Dockerfile
49+
target: openwam
50+
platforms: linux/amd64
51+
load: true
52+
tags: ${{ env.OPENWAM_IMAGE }}
53+
build-args: VCS_REF=${{ github.sha }}
54+
cache-from: type=gha,scope=openwam-cu128
55+
cache-to: type=gha,mode=min,scope=openwam-cu128,ignore-error=true
56+
- name: Check dependencies and run container CPU tests
57+
run: make docker-check
58+
- name: Check Compose isolation, runtime users, indexes, mounts and offline delivery
59+
run: make docker-integration-check PYTHON=/usr/bin/python3

‎Makefile‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ help:
1414
@echo "make check - run compile checks and the core test suite"
1515
@echo "make all - lint + core test"
1616
@echo "make clean - remove Python cache files"
17+
@$(MAKE) --no-print-directory docker-help
1718

1819
test:
1920
$(PYTHON) -m pytest -q -m "not gpu" $(CORE_TEST_ARGS)
@@ -25,14 +26,14 @@ tri-system-smoke:
2526
$(PYTHON) -m pytest -q $(TRI_SYSTEM_SMOKE_ARGS)
2627

2728
lint:
28-
$(PYTHON) -m ruff check openwam/ scripts/ tests/
29+
$(PYTHON) -m ruff check openwam/ scripts/ tests/ docker/
2930

3031
format:
31-
$(PYTHON) -m ruff format openwam/ scripts/ tests/
32-
$(PYTHON) -m ruff check --fix openwam/ scripts/ tests/
32+
$(PYTHON) -m ruff format openwam/ scripts/ tests/ docker/
33+
$(PYTHON) -m ruff check --fix openwam/ scripts/ tests/ docker/
3334

3435
compile:
35-
$(PYTHON) -m compileall openwam scripts tests
36+
$(PYTHON) -m compileall openwam scripts tests docker
3637

3738
check: compile test
3839

@@ -41,3 +42,6 @@ all: lint test
4142
clean:
4243
find . -name "__pycache__" -type d -prune -exec rm -rf {} +
4344
find . -name "*.pyc" -delete
45+
46+
# Keep the default target and native recipes here; Docker commands live together.
47+
include docker/docker.mk

‎README.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,18 @@ All architectures are selected via `configs/model/<framework>.yaml` with `archit
9595

9696
## Installation
9797

98+
Choose one installation route:
99+
100+
| Route | Use it when you want to… |
101+
|---|---|
102+
| [Docker](assets/openwam_usage_docs/docker.md) | run a CUDA policy server, train, develop in a container, or deploy to an offline GPU host |
103+
| [Native Python environment](#native-python-environment) | manage Python and CUDA dependencies directly on your machine |
104+
105+
Docker includes the project's Python environment. GPU hosts still need an
106+
NVIDIA driver and NVIDIA Container Toolkit; see the guide's host requirements.
107+
108+
### Native Python environment
109+
98110
Create an environment with **conda**:
99111

100112
```bash
@@ -390,6 +402,7 @@ entry points for using and extending the repository:
390402

391403
| Guide | Use it when you want to… |
392404
|---|---|
405+
| [Docker and offline deployment](assets/openwam_usage_docs/docker.md) | build an image, run a first policy, train, develop with Compose, or deploy to an offline GPU host |
393406
| [Training and deployment](assets/openwam_usage_docs/train-and-deploy.md) | choose a model/dataloader, prepare assets, train, fine-tune, resume, or deploy a policy |
394407
| [Architecture extension](assets/openwam_usage_docs/architecture-extension.md) | extend a video, visual, VLM, action backbone, or WAM architecture |
395408
| [Benchmark integration](assets/openwam_usage_docs/benchmark-integration.md) | extend a dataloader and connect a benchmark client to the WebSocket protocol |

‎assets/openwam_usage_docs/docker-validation.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Docker validation scope
2+
3+
Results apply to the named revision/image and environment. Use
4+
[docker.md](docker.md) for setup and test commands. No Cosmos or LIBERO assets
5+
were downloaded for the 2026-09-11 checks below.
6+
7+
## Current checks (2026-09-11)
8+
9+
The standard Wan image was built from `24d88ec6712f53e6c925d8ada7a50badbfbd6811`.
10+
Its local image ID was
11+
`sha256:4a7059711b90df4fd0c8ef757fa675d6061b48ebe91bfcbbaf1d5081ff3952d5`.
12+
13+
| Check | Result and scope |
14+
| --- | --- |
15+
| Build and CPU checks | Actual CUDA image built; pip, serve CLI and Ruff passed. **1,990 CPU tests** passed; **12 Docker integration tests** passed separately. |
16+
| Additional CPU tests | All **38** non-GPU tests in `test_tri_system_smoke.py`, excluded by `docker-check`, passed separately. |
17+
| Dependency locking | `make docker-lock` resolved 118 packages and reproduced the committed file exactly in an isolated checkout. `--upgrade` resolved/wrote successfully; three changed packages were installed in a disposable container and passed pip/Ruff and 1,990 CPU tests. The repository lock stayed unchanged; no full upgraded-image build was claimed. |
18+
| Full offline delivery | The complete 9 GB compressed CUDA bundle was exported using schema 3, transferred to the offline H100 and loaded with its bundled importer. Checksums and image layers/config matched across Docker 29.8/containerd and 28.1.1/overlay2. |
19+
| Runtime compatibility | Real GPU compiler/backward/FusedAdam checks passed via `docker exec` before and after restart with `OPENWAM_CUDA_COMPAT=1`. |
20+
| GPU regressions | **7** lightweight GPU-related cases passed with the RAS adjustment below: five use real CUDA tensors and two use simulated dispatch inputs. These cover attention dispatch, train/deploy consistency and RoPE migration without downloaded checkpoints. |
21+
| Serving | Existing RoboTwin checkpoint: container-local bridge health/prediction/reset passed. Host-to-bridge access timed out on this server. The documented host-network overlay passed real ping/prediction/reset from the development machine over an SSH tunnel, including `OPENWAM_SERVER_URL`; predictions contained 20 action values. |
22+
| Training/resume | Four H100s and four existing real RoboTwin episodes: initial training used the source-mounted development shell entrypoint; a new image-only container restored completed step-10 state, ran steps 11–20 and exited 0. Both phases explicitly used `NCCL_RAS_ENABLE=0` and `NCCL_NVLS_ENABLE=0`. |
23+
24+
The H100 host used driver 550.54.14, Toolkit 1.18.1 and kernel 5.4.0.
25+
Training used 583 timesteps / 579 windows, about 6.02B trainable parameters,
26+
BF16, ZeRO-2, batch size 1 per GPU, gradient checkpointing, CPU initialization
27+
and no optimizer offload. Four optimizer shards and four RNG files were
28+
verified; full state was 102.3 GiB. Final weights were 23.1 GiB and differed
29+
from step 10; losses and gradient norms were finite. Debug mode used constant LR.
30+
31+
**Logging limitation:** the first container completed unsaved step 11 before
32+
receiving the stop signal. Resume correctly restarted at 11, but the append-only
33+
CSV retained both step-11 records. The strict no-duplicate-log assertion failed;
34+
an independent artifact verifier confirmed recovery while retaining this
35+
limitation. This does not establish clean, deduplicated metrics after rollback.
36+
37+
## RAS shutdown fix found by GPU testing
38+
39+
With NSS user/group lookup preloaded, both GPU workers completed the smoke
40+
calculations, then rank 0 exited with SIGSEGV. Disabling NVLS or enabling CUDA
41+
compatibility alone did not fix it. Disabling either NSS preload or NCCL RAS
42+
allowed a clean exit. The image now retains NSS and defaults `NCCL_RAS_ENABLE=0`.
43+
44+
The locally rebuilt fix image `openwam:24d88ec-ras-fix`, ID
45+
`sha256:ab59032106f82f8f7df914298367fc0d6e5e38025a765f869c131460e7cc034b`,
46+
passed pip/Ruff, 1,990 CPU tests and 12 Docker integration tests. The actual
47+
rebuilt image was then transferred to H100 over an SSH-forwarded local registry;
48+
its filesystem layers and runtime configuration matched the local build. With
49+
no RAS override, the default two-GPU smoke completed BF16, compiled backward,
50+
FusedAdam and NCCL all-reduce, and the launcher exited **0**. This verifies the
51+
image default, including process shutdown. Earlier GPU regression and training
52+
checks used an equivalent ENV-only image or an explicit RAS setting.
53+
54+
## CI and remaining scope
55+
56+
The original revision's [PR Docker CI](https://github.com/OpenWAM-Official/OpenWAM/actions/runs/34576398602)
57+
passed. Its [push run](https://github.com/OpenWAM-Official/OpenWAM/actions/runs/34576395292)
58+
first failed during Ubuntu index download with `Hash Sum mismatch`, before tests;
59+
its second attempt passed. These CI runs predate the RAS fix.
60+
61+
Not validated: Cosmos image/GPU execution, LIBERO download-to-training workflow,
62+
full-dataset or long-running training, convergence/benchmark quality, changing-LR
63+
scheduler recovery, bitwise equality to uninterrupted training, multi-node/RDMA,
64+
or a broader GPU/driver matrix. CPU CI cannot establish these properties.
65+
66+
The earlier `9ef0cd9` H100 acceptance and local review-image records remain in
67+
Git history; [its original CUDA CI](https://github.com/KraHsu/OpenWAM-Official/actions/runs/34466639632)
68+
also passed. Two H100s ran out of memory for the historical training setup;
69+
four-GPU initialization required the host-specific NVLS workaround.

0 commit comments

Comments
 (0)