Skip to content

Commit fc835a3

Browse files
jamesli124claude
andcommitted
build(docker): add container image for the toolkit CLI
Packages FinaleToolkit with a pinned Python and its compiled dependencies (pysam, numba, pyBigWig, py2bit) so it runs identically on a laptop, in CI, and on a cloud instance without reproducing the environment by hand. Two-stage build: a builder stage produces a wheel, and a clean runtime stage installs it, keeping compilers and build tooling out of the final image. A third `test` stage adds pytest as a strict superset of the runtime image; the ~91MB of fixtures under tests/ are mounted at run time rather than baked in. .git is excluded from the build context (~200MB), so setuptools-scm cannot derive the version inside the container. docker/build.sh resolves it on the host and passes it as SETUPTOOLS_SCM_PRETEND_VERSION, preferring setuptools-scm and falling back to a git-describe implementation of guess-next-dev. The build fails rather than shipping an image that reports itself as 0.0.0.dev0+unknown. docker/verify.py runs as a build step and guards three failure modes that would otherwise ship silently: the fallback version, missing package data under frag/data and genome/data, and compiled wheels that do not load. Targets linux/amd64 by default, since py2bit and pyBigWig do not reliably publish prebuilt ARM Linux wheels. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 3ca99fd commit fc835a3

5 files changed

Lines changed: 449 additions & 0 deletions

File tree

.dockerignore

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Paths excluded from the Docker build context -- the daemon never receives
2+
# them. The Dockerfile only COPYs pyproject.toml, MANIFEST.in, README.md,
3+
# LICENSE and src/, so this is about keeping builds fast, not correctness.
4+
5+
# ~200MB of history. setuptools-scm would normally read the version from it;
6+
# docker/build.sh passes the version in as a build arg instead.
7+
.git
8+
.gitignore
9+
.github
10+
11+
# ~91MB of BAM/CRAM/2bit fixtures. Mounted at run time by the "test" stage
12+
# rather than baked into the image.
13+
tests
14+
15+
# Not needed to build or run the package.
16+
docs
17+
scripts
18+
.readthedocs.yaml
19+
CHANGELOG.md
20+
RELEASING.md
21+
22+
# Build and test artifacts.
23+
build
24+
dist
25+
*.egg-info
26+
**/__pycache__
27+
**/*.pyc
28+
.pytest_cache
29+
.ruff_cache
30+
.coverage*
31+
32+
# Local editor / OS cruft.
33+
.vscode
34+
.claude
35+
**/.DS_Store
36+
37+
# Written by setuptools-scm during the build. A stale copy from a local
38+
# editable install would otherwise be copied in and shadow the real version.
39+
src/finaletoolkit/_version.py

Dockerfile

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# syntax=docker/dockerfile:1
2+
3+
# Two-stage build. The first stage has the tooling needed to produce a wheel;
4+
# the second installs that wheel into a clean image, so the compilers and
5+
# build-time packages never ship in the final artifact.
6+
#
7+
# Build with docker/build.sh, which derives the version from git tags and
8+
# passes it in. See docker/README.md.
9+
10+
ARG PYTHON_VERSION=3.12
11+
12+
13+
# --- Stage 1: build the wheel ------------------------------------------------
14+
FROM python:${PYTHON_VERSION}-slim AS builder
15+
16+
# setuptools-scm normally derives the version from git history, but .git is
17+
# excluded from the build context (it is ~200MB, and shipping it to the daemon
18+
# on every build is slow). The version is passed in instead. Without this,
19+
# setuptools-scm silently falls back to "0.0.0.dev0+unknown" -- the runtime
20+
# stage asserts against exactly that.
21+
ARG SETUPTOOLS_SCM_PRETEND_VERSION
22+
ENV SETUPTOOLS_SCM_PRETEND_VERSION=${SETUPTOOLS_SCM_PRETEND_VERSION}
23+
24+
WORKDIR /src
25+
26+
RUN --mount=type=cache,target=/root/.cache/pip \
27+
pip install build
28+
29+
# MANIFEST.in and README.md are referenced by the build (readme is declared in
30+
# pyproject.toml), so the build fails without them even though only src/ ends
31+
# up in the wheel.
32+
COPY pyproject.toml MANIFEST.in README.md LICENSE ./
33+
COPY src/ ./src/
34+
35+
RUN --mount=type=cache,target=/root/.cache/pip \
36+
python -m build --wheel --outdir /dist
37+
38+
39+
# --- Stage 2: runtime --------------------------------------------------------
40+
FROM python:${PYTHON_VERSION}-slim AS runtime
41+
42+
LABEL org.opencontainers.image.title="FinaleToolkit" \
43+
org.opencontainers.image.description="Extract fragmentation features of cell-free DNA from paired-end sequencing data." \
44+
org.opencontainers.image.source="https://github.com/epifluidlab/FinaleToolkit" \
45+
org.opencontainers.image.documentation="https://epifluidlab.github.io/FinaleToolkit/" \
46+
org.opencontainers.image.licenses="MIT"
47+
48+
# numba and matplotlib both want a writable cache/config directory and fall
49+
# back to $HOME, which the non-root user below cannot always write to
50+
# (e.g. when the image is run with an arbitrary --user). Point them at /tmp,
51+
# which is writable regardless.
52+
#
53+
# PYTHONDONTWRITEBYTECODE is deliberately *not* set: pip precompiles .pyc
54+
# files at install time, and keeping them means the CLI does not re-parse
55+
# scipy/numba/matplotlib source on every invocation. This is a short-lived
56+
# process started many times, so startup cost matters more than image size.
57+
ENV NUMBA_CACHE_DIR=/tmp/numba-cache \
58+
MPLCONFIGDIR=/tmp/matplotlib \
59+
PYTHONUNBUFFERED=1
60+
61+
# Bind-mounting the wheel rather than COPYing it keeps it out of the image's
62+
# layers -- the installed package is all we need.
63+
RUN --mount=type=bind,from=builder,source=/dist,target=/dist \
64+
--mount=type=cache,target=/root/.cache/pip \
65+
pip install /dist/*.whl
66+
67+
# Two smoke checks that catch the failure modes that would otherwise ship
68+
# silently:
69+
#
70+
# 1. Version. Without SETUPTOOLS_SCM_PRETEND_VERSION the package falls back to
71+
# "0.0.0.dev0+unknown" and the image is untraceable.
72+
# 2. Package data. The .tsv/.gz files under frag/data and genome/data are
73+
# pulled in by [tool.setuptools.package-data]; if that ever stops working,
74+
# the CLI installs cleanly and then fails at run time on the subcommands
75+
# that need them.
76+
# 3. Imports. Loading the CLI module confirms the compiled wheels (pysam,
77+
# numba, pyBigWig, py2bit) actually load on this base image.
78+
#
79+
# Bind-mounted rather than COPYed so the script leaves no layer behind.
80+
RUN --mount=type=bind,source=docker/verify.py,target=/tmp/verify.py \
81+
python /tmp/verify.py
82+
83+
RUN useradd --create-home --uid 1000 finaletoolkit \
84+
&& mkdir -p /data \
85+
&& chown finaletoolkit:finaletoolkit /data
86+
87+
USER finaletoolkit
88+
89+
# Conventional mount point for input BAM/CRAM files and outputs:
90+
# docker run -v /host/path:/data <image> <subcommand> /data/sample.bam ...
91+
WORKDIR /data
92+
93+
# ENTRYPOINT makes the image behave as the CLI itself, so `docker run <image>
94+
# frag-length-bins ...` works the same way the installed command does.
95+
ENTRYPOINT ["finaletoolkit"]
96+
CMD ["--help"]
97+
98+
99+
# --- Stage 3: test -----------------------------------------------------------
100+
# A strict superset of the runtime image: same environment, plus pytest. The
101+
# 91MB of fixtures under tests/ are mounted at run time rather than baked in.
102+
#
103+
# docker build --target test -t finaletoolkit:test .
104+
# docker run --rm -v "$PWD/tests:/app/tests:ro" --entrypoint pytest \
105+
# finaletoolkit:test
106+
#
107+
# The ENTRYPOINT is deliberately left as-is so this image still works as a
108+
# plain CLI if it is built by accident (it is the last stage, so a bare
109+
# `docker build .` lands here).
110+
FROM runtime AS test
111+
112+
USER root
113+
RUN --mount=type=cache,target=/root/.cache/pip \
114+
pip install pytest pytest-cov
115+
116+
# pytest reads [tool.pytest.ini_options] from pyproject.toml, which sets
117+
# testpaths = ["tests"]; this also makes /app the rootdir. It has to be
118+
# writable by the non-root user, since pytest writes .pytest_cache there and
119+
# --cov writes .coverage.
120+
RUN mkdir -p /app && chown finaletoolkit:finaletoolkit /app
121+
WORKDIR /app
122+
COPY --chown=finaletoolkit:finaletoolkit pyproject.toml ./
123+
124+
USER finaletoolkit

docker/README.md

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# Container image
2+
3+
A container image bundles FinaleToolkit with a pinned Python and all of its
4+
compiled dependencies (numpy, pysam, numba, pyBigWig, py2bit), so it runs
5+
identically on a laptop, in CI, and on a cloud instance without anyone having
6+
to reproduce the environment by hand.
7+
8+
## Build
9+
10+
```sh
11+
docker/build.sh
12+
```
13+
14+
This tags `finaletoolkit:<version>` and `finaletoolkit:latest`, where
15+
`<version>` comes from git tags via setuptools-scm — the same version the wheel
16+
would get.
17+
18+
Why a script rather than a bare `docker build`: `.git` is excluded from the
19+
build context (it is ~200MB), so setuptools-scm cannot derive the version
20+
inside the container. The script resolves it and passes it in as a build arg.
21+
Building by hand works too, as long as you supply it:
22+
23+
```sh
24+
docker build --build-arg SETUPTOOLS_SCM_PRETEND_VERSION=1.1.0 \
25+
--target runtime -t finaletoolkit:1.1.0 .
26+
```
27+
28+
The build fails rather than producing an image that reports itself as
29+
`0.0.0.dev0+unknown`.
30+
31+
Overrides, all optional:
32+
33+
| Variable | Default | Purpose |
34+
| --- | --- | --- |
35+
| `FTK_VERSION` | derived from git | Set the version explicitly |
36+
| `FTK_IMAGE` | `finaletoolkit` | Image name |
37+
| `FTK_PLATFORM` | `linux/amd64` | Target architecture |
38+
39+
`linux/amd64` is the default because py2bit and pyBigWig do not reliably
40+
publish prebuilt ARM Linux wheels; on ARM they may have to compile from source.
41+
On Apple Silicon the resulting image runs under emulation — correct, but slow.
42+
43+
## Run
44+
45+
The image behaves as the `finaletoolkit` command itself:
46+
47+
```sh
48+
docker run --rm finaletoolkit:latest --help
49+
docker run --rm finaletoolkit:latest --version
50+
```
51+
52+
A container has its own filesystem that is discarded on exit, so input and
53+
output have to be bind-mounted from the host. `/data` is the conventional
54+
mount point and the image's working directory:
55+
56+
```sh
57+
docker run --rm -v "/path/to/data:/data" finaletoolkit:latest \
58+
frag-length-bins /data/sample.bam -o /data/output.tsv
59+
```
60+
61+
Paths in the command are container paths, not host paths — `/data/sample.bam`
62+
refers to `/path/to/data/sample.bam` on the host.
63+
64+
The container runs as a non-root user (uid 1000). On Linux, output files land
65+
on the host owned by that uid; add `--user "$(id -u):$(id -g)"` if you need
66+
them owned by you. Docker Desktop on macOS handles this mapping already.
67+
68+
## Test
69+
70+
The `test` stage is the runtime image plus pytest. The ~91MB of fixtures under
71+
`tests/` are mounted at run time rather than baked into the image:
72+
73+
```sh
74+
docker/build.sh --target test
75+
docker run --rm -v "$PWD/tests:/app/tests:ro" \
76+
--entrypoint pytest finaletoolkit:test
77+
```
78+
79+
Tests write through pytest's `tmp_path` fixture, so the mount can stay
80+
read-only.
81+
82+
Passing tests here is a stronger signal than passing on the host: it confirms
83+
the packaged environment is correct, not just that the code imports.
84+
85+
## What is in the image
86+
87+
Two stages. The first installs the build tooling and produces a wheel; the
88+
second installs that wheel into a clean `python:3.12-slim` base, so compilers
89+
and build-time packages never ship in the final artifact.
90+
91+
The Python version is pinned to one version (3.12), unlike CI, which tests a
92+
3.10–3.13 matrix. That is intentional: the matrix proves the library is
93+
portable, while the image is a single reproducible deployment target.
94+
95+
`NUMBA_CACHE_DIR` and `MPLCONFIGDIR` are pointed at `/tmp` because numba and
96+
matplotlib both want a writable cache directory and default to `$HOME`, which
97+
is not writable under an arbitrary `--user`.

docker/build.sh

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Build the FinaleToolkit container image.
4+
#
5+
# .git is excluded from the build context, so setuptools-scm cannot derive the
6+
# version inside the container. This script resolves it here and passes it in.
7+
#
8+
# Usage:
9+
# docker/build.sh # build the runtime image
10+
# docker/build.sh --target test # build the test image instead
11+
# FTK_VERSION=1.1.0 docker/build.sh
12+
#
13+
# Any extra arguments are forwarded to `docker build`.
14+
15+
set -euo pipefail
16+
17+
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
18+
cd "$repo_root"
19+
20+
image="${FTK_IMAGE:-finaletoolkit}"
21+
platform="${FTK_PLATFORM:-linux/amd64}"
22+
23+
# Mirrors setuptools-scm's default version_scheme ("guess-next-dev") combined
24+
# with the local_scheme = "no-local-version" set in pyproject.toml:
25+
#
26+
# exactly on tag v1.1.0, clean -> 1.1.0
27+
# 4 commits past v1.1.0 -> 1.1.1.dev4
28+
#
29+
# Only used when setuptools-scm is not importable. If the version_scheme in
30+
# pyproject.toml ever changes, this needs to change with it.
31+
derive_version_from_git() {
32+
local describe tag distance dirty base
33+
describe="$(git describe --tags --long --dirty 2>/dev/null)" || return 1
34+
35+
dirty=""
36+
if [[ "$describe" == *-dirty ]]; then
37+
dirty="yes"
38+
describe="${describe%-dirty}"
39+
fi
40+
41+
# v1.1.0-4-g3ca99fd -> tag=v1.1.0, distance=4
42+
tag="${describe%-*-g*}"
43+
distance="${describe%-g*}"
44+
distance="${distance##*-}"
45+
46+
base="${tag#v}"
47+
48+
if [[ "$distance" == "0" && -z "$dirty" ]]; then
49+
printf '%s\n' "$base"
50+
return 0
51+
fi
52+
53+
# Bump the last numeric component: 1.1.0 -> 1.1.1
54+
local prefix last
55+
last="${base##*.}"
56+
prefix="${base%.*}"
57+
if [[ "$last" =~ ^[0-9]+$ ]]; then
58+
printf '%s.%s.dev%s\n' "$prefix" "$((last + 1))" "$distance"
59+
else
60+
return 1
61+
fi
62+
}
63+
64+
# setuptools-scm is authoritative when available; the git fallback keeps this
65+
# script usable in an environment that only has the runtime dependencies.
66+
version="${FTK_VERSION:-}"
67+
if [[ -z "$version" ]]; then
68+
version="$(python -m setuptools_scm 2>/dev/null || true)"
69+
fi
70+
if [[ -z "$version" ]]; then
71+
version="$(derive_version_from_git || true)"
72+
[[ -n "$version" ]] && echo "note: setuptools-scm unavailable; derived ${version} from git" >&2
73+
fi
74+
if [[ -z "$version" ]]; then
75+
cat >&2 <<'EOF'
76+
error: could not determine the package version.
77+
78+
setuptools-scm is not importable and the version could not be derived from git
79+
(no tags reachable, or not a git checkout). Either:
80+
81+
pip install setuptools-scm
82+
83+
or set it explicitly:
84+
85+
FTK_VERSION=1.1.0 docker/build.sh
86+
EOF
87+
exit 1
88+
fi
89+
90+
# Default to the runtime stage. Without this a bare `docker build` would build
91+
# the last stage in the Dockerfile, which is the larger test image.
92+
target="runtime"
93+
args=()
94+
while [[ $# -gt 0 ]]; do
95+
case "$1" in
96+
--target)
97+
target="$2"
98+
shift 2
99+
;;
100+
--target=*)
101+
target="${1#*=}"
102+
shift
103+
;;
104+
*)
105+
args+=("$1")
106+
shift
107+
;;
108+
esac
109+
done
110+
111+
# Non-runtime stages get a suffixed tag so that building the test image does
112+
# not overwrite the runtime image's tags.
113+
tags=()
114+
if [[ "$target" == "runtime" ]]; then
115+
tags+=(--tag "${image}:${version}" --tag "${image}:latest")
116+
else
117+
tags+=(--tag "${image}:${version}-${target}" --tag "${image}:${target}")
118+
fi
119+
120+
echo "Building ${image} ${version} (target=${target}, platform=${platform})"
121+
122+
set -x
123+
docker build \
124+
--platform "$platform" \
125+
--target "$target" \
126+
--build-arg "SETUPTOOLS_SCM_PRETEND_VERSION=${version}" \
127+
"${tags[@]}" \
128+
"${args[@]+"${args[@]}"}" \
129+
.

0 commit comments

Comments
 (0)