Skip to content

Commit a52b433

Browse files
committed
Add CNTK rescue assets and DPPOP container scaffold
1 parent 9e33d65 commit a52b433

5 files changed

Lines changed: 452 additions & 0 deletions

File tree

‎.dockerignore‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
.git
2+
.github
3+
.vs
4+
.vscode
5+
.fake
6+
**/bin
7+
**/obj
8+
artifacts/*
9+
!artifacts/legacy-runtime/
10+
!artifacts/legacy-runtime/imlp-1.0.0/
11+
!artifacts/legacy-runtime/imlp-1.0.0/legacy-runtime.tar.gz
12+
!artifacts/legacy-runtime/imlp-1.0.0/runtime-manifest.json
13+
!artifacts/legacy-runtime/imlp-1.0.0/SHA256SUMS

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -374,6 +374,8 @@ MigrationBackup/
374374

375375
# Ionide (cross platform F# VS Code tools) working folder
376376
.ionide/
377+
.dotnet/
378+
src/BioFSharp.ML/Resources/*.fasta
377379

378380
# Fody - auto-generated XML schema
379381
FodyWeavers.xsd

‎Dockerfile‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
FROM mcr.microsoft.com/dotnet/sdk:8.0-jammy-amd64 AS build
2+
3+
WORKDIR /src
4+
COPY . .
5+
6+
RUN dotnet publish src/DPPOP.CLI/DPPOP.CLI.fsproj -c Release -o /app/publish
7+
8+
FROM mcr.microsoft.com/dotnet/runtime:8.0-jammy-amd64
9+
10+
RUN apt-get update -y \
11+
&& apt-get install -y --no-install-recommends libnuma1 \
12+
&& rm -rf /var/lib/apt/lists/*
13+
14+
COPY artifacts/legacy-runtime/imlp-1.0.0/legacy-runtime.tar.gz /tmp/legacy-runtime.tar.gz
15+
RUN tar -xzf /tmp/legacy-runtime.tar.gz -C / \
16+
&& rm -f /tmp/legacy-runtime.tar.gz
17+
18+
ENV PATH="/usr/local/cntk/cntk/lib:/usr/local/mpi/bin:${PATH}"
19+
ENV LD_LIBRARY_PATH="/usr/local/cntk/cntk/dependencies/lib:/usr/local/cntk/cntk/lib:/usr/local/mpi/lib:${LD_LIBRARY_PATH}"
20+
21+
WORKDIR /data
22+
COPY --from=build /app/publish ./
23+
24+
ENTRYPOINT ["dotnet", "DPPOP.CLI.dll"]

‎plans/rescue_modernize.md‎

Lines changed: 273 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,273 @@
1+
# Rescue and Modernization Plan for iMLP and DPPOP
2+
3+
## Ground truth
4+
5+
The following points are based on inspection of the current repositories and the local Docker image state.
6+
7+
- `iMLP` is a `net5.0` packaged tool that still loads CNTK directly via `CNTK.CPUOnly 2.7`.
8+
- `iMLP` already contains both CNTK and ONNX model assets in source control.
9+
- The current `iMLP` Docker build relies on downloading CNTK from a URL that is no longer dependable.
10+
- The local Docker image `csbdocker/imlp:1.0.0` is available and can serve as a known-good rescue source.
11+
- The image contains working CNTK and OpenMPI runtime trees under `/usr/local/cntk` and `/usr/local/mpi`.
12+
- The actual CNTK dependency path in the image is `/usr/local/cntk/cntk/dependencies/lib`.
13+
- `BioFSharp.ML` is currently a library, not a packaged CLI tool.
14+
- DPPOP prediction logic already exists in `src/BioFSharp.ML/DPPOP.fs`.
15+
- The current DPPOP entrypoint is script-based via `src/BioFSharp.ML/Scripts/DPPOP.fsx`.
16+
- `BioFSharp.ML` already pins SDK `8.0.100` in `global.json`.
17+
- `BioFSharp.ML` still references `CNTK.CPUOnly 2.8.0-rc0.dev20200201`.
18+
19+
## Goals
20+
21+
1. Rescue the legacy CNTK runtime so both projects remain buildable and deployable.
22+
2. Keep `iMLP` operational while moving it to a more modern .NET toolchain.
23+
3. Turn DPPOP into a proper CLI and containerize it in the same operational style as `iMLP`.
24+
4. Optionally evaluate ONNX export for DPPOP.
25+
5. Optionally migrate both tools from CNTK inference to ONNX Runtime once parity is proven.
26+
27+
## Phase 1: Rescue the CNTK runtime
28+
29+
The first priority is to preserve a reproducible CNTK runtime independent of external URLs.
30+
31+
### Deliverables
32+
33+
- A rescued runtime artifact extracted from `csbdocker/imlp:1.0.0`.
34+
- Checksums and minimal documentation for the rescued payload.
35+
- A repeatable extraction script or procedure.
36+
37+
### Scope of rescue
38+
39+
Preserve the working runtime directories from the local Docker image:
40+
41+
- `/usr/local/cntk/cntk/lib`
42+
- `/usr/local/cntk/cntk/dependencies/lib`
43+
- `/usr/local/mpi/lib`
44+
45+
At minimum, preserve:
46+
47+
- CNTK managed and native libraries
48+
- CNTK dependency libraries
49+
- OpenMPI runtime libraries
50+
- Any path or loader assumptions required at runtime
51+
52+
### Recommended storage strategy
53+
54+
Do not rely on rebuilding the runtime from upstream downloads.
55+
56+
Instead:
57+
58+
- archive the rescued binaries as an internal release artifact, package archive, or immutable binary backup
59+
- record the image digest and extraction source
60+
- store SHA256 checksums for each published archive
61+
- include a short `README` describing required environment variables and expected directory layout
62+
63+
### Notes
64+
65+
- The local image is the strongest current recovery anchor.
66+
- The current Dockerfile should be treated as historical documentation, not as the sole source of truth.
67+
68+
## Phase 2: Stabilize and modernize iMLP while staying on CNTK
69+
70+
`iMLP` should first be made reproducible from source without changing the inference backend.
71+
72+
### Objectives
73+
74+
- stop depending on the dead CNTK download URL
75+
- keep existing model behavior unchanged
76+
- move off the .NET 5-era build environment
77+
78+
### Proposed steps
79+
80+
1. Replace CNTK download logic in the Docker build with the rescued runtime artifact.
81+
2. Correct the CNTK dependency path assumptions so the container reflects the actual working layout.
82+
3. Build `iMLP` from source inside Docker instead of depending primarily on `dotnet tool install imlp --version ...`.
83+
4. Add `global.json` to pin the SDK used for `iMLP`.
84+
5. Retarget `iMLP` from `net5.0` to `net8.0` first.
85+
6. Keep CNTK model loading unchanged during this phase.
86+
7. Add a smoke test that runs `imlp` on a known short sequence and checks for non-empty output.
87+
88+
### Why `net8.0` first
89+
90+
- `BioFSharp.ML` is already aligned to .NET 8 SDK tooling.
91+
- It reduces the number of moving parts compared to jumping directly to a newer target.
92+
- It gives a supported LTS baseline before any runtime migration work.
93+
94+
### Acceptance criteria
95+
96+
- `iMLP` builds from source in a container without external CNTK downloads.
97+
- The rescued CNTK runtime is sufficient for prediction execution.
98+
- The CLI output for a small golden input remains unchanged or acceptably close.
99+
100+
## Phase 3: Containerize DPPOP as a real CLI
101+
102+
DPPOP should become a proper executable rather than remaining script-driven.
103+
104+
### Current state
105+
106+
- core prediction logic already exists in the library
107+
- the repo does not currently expose DPPOP as a tool project
108+
- the current script is useful as a reference, but not as the deployment artifact
109+
110+
### Proposed implementation
111+
112+
Create a new executable project, for example:
113+
114+
- `src/DPPOP.CLI`
115+
116+
This project should:
117+
118+
- reference `BioFSharp.ML`
119+
- expose a stable command-line interface
120+
- reuse `DPPOP.Prediction.scoreDppopPlant`
121+
- reuse `DPPOP.Prediction.scoreDppopNonPlant`
122+
- optionally support a custom model path
123+
124+
### Suggested CLI shape
125+
126+
- `--proteome <fasta>`
127+
- `--proteins-of-interest <fasta>`
128+
- `--model plant|nonplant|custom`
129+
- `--custom-model <path>` when applicable
130+
- `--output <tsv>`
131+
132+
### Suggested output schema
133+
134+
One peptide per row, with at least:
135+
136+
- `ProteinId`
137+
- `Sequence`
138+
- `PredictionScore`
139+
- `Distinct`
140+
141+
### Containerization approach
142+
143+
Containerize DPPOP in the same operational pattern as `iMLP`:
144+
145+
- same rescued CNTK/OpenMPI runtime base
146+
- same `/data` working directory convention
147+
- same bind-mount based workflow
148+
- same expectation that FASTA inputs are mounted from host paths
149+
150+
### Acceptance criteria
151+
152+
- DPPOP can be run non-interactively from a container
153+
- the CLI accepts FASTA inputs and writes deterministic TSV output
154+
- the container no longer depends on ad hoc F# script execution
155+
156+
## Phase 4: Unify runtime and image layout
157+
158+
Once both tools run on the rescued CNTK stack, consolidate the container setup.
159+
160+
### Recommended structure
161+
162+
Build a shared base image or common Docker stage that contains:
163+
164+
- rescued CNTK runtime
165+
- rescued OpenMPI runtime
166+
- required environment variables
167+
168+
Then build:
169+
170+
- an `iMLP` image on top of that base
171+
- a `DPPOP` image on top of that base
172+
173+
### Benefits
174+
175+
- one maintained legacy runtime layer
176+
- less duplication
177+
- easier recovery if future registry URLs disappear
178+
- clearer operational documentation
179+
180+
## Optional Phase 5: Export DPPOP models to ONNX
181+
182+
This is worth exploring, but should be treated as a separate validation track rather than part of the rescue path.
183+
184+
### Why this is optional
185+
186+
- the core rescue problem is runtime preservation, not model conversion
187+
- model export may fail or produce numerically different results
188+
- it should not block containerization
189+
190+
### Proposed work
191+
192+
1. Identify the exact CNTK graph characteristics of the DPPOP models.
193+
2. Attempt export of the plant and non-plant models to ONNX.
194+
3. Build a golden test set of peptide predictions.
195+
4. Compare CNTK vs ONNX outputs for:
196+
- raw scores
197+
- sorted peptide ranking
198+
- normalized relative output
199+
200+
### Acceptance criteria
201+
202+
- ONNX export succeeds for both models
203+
- output parity is within a pre-defined tolerance
204+
- no feature preprocessing mismatch is introduced
205+
206+
## Optional Phase 6: Move both projects to ONNX Runtime
207+
208+
This phase should only begin after ONNX parity is proven.
209+
210+
### iMLP
211+
212+
`iMLP` is the easier candidate because ONNX model assets already exist in the repository.
213+
214+
Required work will likely include:
215+
216+
- wiring ONNX Runtime into the existing prediction path
217+
- handling model input shape and output shape explicitly
218+
- verifying whether sidecar `Parameter*` files are required at runtime
219+
- validating predictions against the current CNTK implementation
220+
221+
### DPPOP
222+
223+
DPPOP may be migratable after successful export, but this depends on the exact CNTK graph and preprocessing assumptions.
224+
225+
Required work will likely include:
226+
227+
- ONNX export validation
228+
- a new inference implementation using ONNX Runtime
229+
- parity testing against existing CNTK results
230+
231+
### Acceptance criteria
232+
233+
- CNTK and ONNX outputs match within accepted tolerance
234+
- the modern runtime path is production-ready
235+
- CNTK can remain as a fallback until confidence is high enough to remove it
236+
237+
## Recommended order of execution
238+
239+
1. Rescue and archive the working CNTK/OpenMPI runtime from `csbdocker/imlp:1.0.0`.
240+
2. Make `iMLP` build from source in Docker using the rescued runtime.
241+
3. Retarget `iMLP` to a modern .NET toolchain while keeping CNTK.
242+
4. Add a proper `DPPOP.CLI` executable project.
243+
5. Containerize DPPOP on the same rescued runtime base.
244+
6. Consolidate both tools onto a shared legacy-runtime image layout.
245+
7. Explore DPPOP ONNX export.
246+
8. Only then evaluate full ONNX Runtime migration for one or both projects.
247+
248+
## Risks and constraints
249+
250+
- CNTK is deprecated and should be treated as a legacy dependency to preserve, not extend further than necessary.
251+
- Native dependency loading is the most fragile part of both toolchains.
252+
- Upgrading framework targets and switching inference runtimes in the same change would add unnecessary risk.
253+
- DPPOP currently lacks meaningful automated tests around real prediction behavior.
254+
- Existing Docker metadata and source may not perfectly reflect the live working runtime; the local image is therefore especially valuable.
255+
256+
## Practical next tasks
257+
258+
The next implementation backlog should start with the following concrete tasks:
259+
260+
1. Add a binary rescue procedure for CNTK/OpenMPI based on the local `csbdocker/imlp:1.0.0` image.
261+
2. Patch the `iMLP` Docker build to use the rescued runtime artifact.
262+
3. Add `global.json` and retarget `iMLP` to `net8.0`.
263+
4. Add a minimal container smoke test for `iMLP`.
264+
5. Scaffold `src/DPPOP.CLI` in `BioFSharp.ML`.
265+
6. Define DPPOP CLI arguments and TSV output format.
266+
7. Containerize DPPOP using the same runtime base as `iMLP`.
267+
8. Add golden-result tests for both tools before attempting ONNX migration.
268+
269+
## Assumptions
270+
271+
- The local Docker image `csbdocker/imlp:1.0.0` remains available long enough to extract the rescue artifacts.
272+
- CNTK package restore may still work in some environments, but should not be trusted as the primary recovery mechanism.
273+
- ONNX migration is desirable, but not required to complete the immediate rescue and containerization effort.

0 commit comments

Comments
 (0)