Skip to content

Commit 1b9a3fd

Browse files
lfnothiaslfnothias
authored
Merge pull request #40 from HolobiomicsLab/feat/w4m-mhd-grounded-skills
Ground the MetabolomicsHub and Workflow4Metabolomics skills in their sources
2 parents ca42e29 + 7383ce2 commit 1b9a3fd

13 files changed

Lines changed: 1380 additions & 76 deletions

File tree

‎collections/metabolomics/v2/proposals/skills/galaxy-workflow4metabolomics-reproducible-processing/SKILL.md‎

Lines changed: 85 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
---
22
name: galaxy-workflow4metabolomics-reproducible-processing
3-
description: Use when running an LC-MS or GC-MS preprocessing and statistics workflow
4-
on the Galaxy Workflow4Metabolomics instance, or when a collaborator needs to re-execute
5-
an analysis without installing the toolchain locally.
3+
description: Use when running an LC-MS or GC-MS preprocessing and statistics pipeline
4+
on a Galaxy Workflow4Metabolomics instance, or when a collaborator must re-execute
5+
the analysis without installing the toolchain — the stage order is fixed by the
6+
wrappers' own datatypes, not by convention.
67
license: CC-BY-4.0
78
status: hold
89
metadata:
@@ -13,65 +14,118 @@ metadata:
1314
- LC-MS
1415
- GC-MS
1516
repo_url: https://github.com/workflow4metabolomics/tools-metabolomics
16-
related_skills: []
17+
related_skills:
18+
- w4m-three-table-format-conformance
1719
license_tier: open
1820
provenance_tier: repository
1921
tool_license:
2022
tier: open
2123
requires_ack: false
2224
ref: GPL-3.0
23-
url: https://github.com/workflow4metabolomics/tools-metabolomics/blob/main/LICENSE
25+
url: https://github.com/workflow4metabolomics/tools-metabolomics/blob/master/LICENSE.txt
26+
verified_against:
27+
repo_ref: master
28+
observed: '2026-08-21'
2429
schema_version: 0.2.0
2530
---
2631

2732
# galaxy-workflow4metabolomics-reproducible-processing
2833

29-
Run a metabolomics workflow on a shared Galaxy instance so that the analysis can
30-
be re-executed by someone who has neither the software nor the compute.
34+
Run a metabolomics workflow on a shared Galaxy instance so the analysis can be
35+
re-executed by someone who has neither the software nor the compute.
3136

3237
## When this applies
3338

3439
The barrier to reproducing a metabolomics analysis is rarely the method; it is
35-
the environment. Workflow4Metabolomics packages the common LC-MS and GC-MS
36-
pipeline — preprocessing, normalisation, annotation, univariate and multivariate
37-
statistics — as Galaxy tools on a public instance, so a workflow can be shared as
38-
a document that others execute rather than as instructions they reimplement.
40+
the environment. Workflow4Metabolomics packages the LC-MS, GC-MS and NMR
41+
pipelines — preprocessing, annotation, normalisation, univariate and
42+
multivariate statistics — as Galaxy tools, so a workflow is shared as a document
43+
others execute rather than as instructions they reimplement.
3944

40-
Reach for it when the analysis must outlive the machine it was written on, when a
41-
collaborator cannot install the toolchain, or when a submission requires an
45+
Reach for it when the analysis must outlive the machine it was written on, when
46+
a collaborator cannot install the toolchain, or when a submission requires an
4247
executable record of what was run.
4348

49+
## The stage order is enforced, not conventional
50+
51+
The LC-MS wrappers declare intermediate Galaxy datatypes, and each stage accepts
52+
only the datatype the previous one emits. The chain is therefore readable off
53+
the tool definitions:
54+
55+
```
56+
mzML / mzXML / netCDF / mzData
57+
→ MSnbase readMSData → rdata.msnbase.raw
58+
→ xcms findChromPeaks → rdata.xcms.findchrompeaks
59+
→ xcms refineChromPeaks → rdata.xcms.findchrompeaks (optional)
60+
→ xcms findChromPeaks Merger → rdata.xcms.findchrompeaks (multi-sample)
61+
→ xcms groupChromPeaks → rdata.xcms.group
62+
→ xcms adjustRtime → rdata.xcms.retcor
63+
→ xcms groupChromPeaks → rdata.xcms.group (second pass)
64+
→ xcms fillChromPeaks → rdata.xcms.fillpeaks
65+
→ CAMERA annotate → rdata.camera.* + the three tables
66+
```
67+
68+
Two consequences follow from the datatypes themselves. `adjustRtime` consumes a
69+
grouped object, so correspondence precedes alignment and is then repeated
70+
against the corrected retention times — the second grouping pass is required,
71+
not a refinement. And `fillChromPeaks` accepts only `rdata.xcms.group`, so gap
72+
filling cannot be moved after annotation.
73+
74+
Parameter optimisation sits beside the chain rather than in it: `IPO for
75+
xcmsSet` reads raw files and `IPO for group and retcor` reads the xcms objects,
76+
and both emit parameter tables you feed back into the corresponding step.
77+
78+
Downstream of CAMERA the pipeline works on the three-table format, and
79+
`Check Format` is the entry point to it. Polarity modes are processed separately
80+
and joined with `CAMERA combinexsAnnos`. `Mz(X)ML Shaper` reshapes open formats
81+
into XCMS-readable mz(X)ML — it accepts mzML, mzXML and netCDF only, so vendor
82+
conversion still happens before upload and outside the platform.
83+
4484
## Procedure
4585

46-
1. **Upload the raw data in an open format.** Convert vendor files to mzML first.
86+
1. **Upload raw data in an open format.** Convert vendor files to mzML first.
4787
Uploading vendor formats defers the conversion problem to whoever reruns the
4888
workflow, which defeats the purpose.
49-
2. **Build the sample metadata table before processing**, with one row per file
50-
and explicit columns for class, batch and injection order. Most downstream
51-
failures in this pipeline are metadata failures, and they surface late.
52-
3. **Assemble the workflow from the instance's tool set**, keeping preprocessing,
53-
normalisation and statistics as separate steps rather than one composite. A
54-
step you cannot inspect is a step you cannot defend.
55-
4. **Record every non-default parameter.** The workflow document stores them, but
56-
a reader needs to know which ones were chosen deliberately and why.
57-
5. **Export the workflow and the invocation**, not only the results. The workflow
58-
is the method; the invocation ties it to this dataset and these parameters.
89+
90+
2. **Build the sample metadata table before processing**, one row per file, with
91+
explicit columns for class, batch and injection order. Most downstream
92+
failures in this pipeline are metadata failures and they surface late. The
93+
`xcms get a sampleMetadata file` tool emits the skeleton to fill in.
94+
95+
3. **Keep the stages separate.** Preprocessing, annotation, normalisation and
96+
statistics stay distinct steps rather than one composite. A step you cannot
97+
inspect is a step you cannot defend, and the datatypes above give you the
98+
inspection points for free.
99+
100+
4. **Record every non-default parameter.** The workflow document stores them,
101+
but a reader needs to know which were chosen deliberately and why. Where a
102+
parameter came from IPO, say so and keep the IPO output.
103+
104+
5. **Export the workflow and the invocation**, not only the results. The
105+
workflow is the method; the invocation ties it to this dataset, these
106+
parameters and the tool versions that actually ran.
59107

60108
## Verification
61109

110+
- `xcms process history` summarises what ran; read it rather than trusting the
111+
workflow diagram, which shows what was requested.
62112
- The exported workflow re-runs on the same inputs and yields the same feature
63113
count. A difference means a parameter was not captured or a tool version moved.
64114
- The sample metadata row count matches the uploaded file count.
65-
- The feature table's sample columns match the metadata's sample identifiers
66-
exactly — a silent mismatch here produces statistics on mislabelled groups.
115+
- The three tables agree on identifiers and order before any statistics step.
116+
- Both polarities, if processed, were combined once and not double-counted.
67117

68118
## Limitations
69119

70-
- Public instances impose quotas on storage and runtime; a large study may need a
71-
local or institutional Galaxy rather than the shared one.
72-
- Tool versions on the instance change over time. A workflow exported today may
73-
resolve to different tool versions later, so the invocation record — which pins
74-
versions — matters more than the workflow alone.
120+
- Public instances impose quotas on storage and runtime; a large study may need
121+
an institutional Galaxy rather than the shared one.
122+
- Tool versions on the instance change over time, and most wrappers version
123+
themselves against the underlying R package. A workflow exported today may
124+
resolve to different versions later, so the invocation record matters more
125+
than the workflow alone.
75126
- The available tools bound the method. A step the instance does not provide
76127
cannot be inserted without deploying a tool, which is an administrative task
77128
rather than an analytical one.
129+
- The datatype chain above is the LC-MS line. The NMR tools and the
130+
isotope-labelling and flux tools in the same repository form separate chains
131+
that share only the three-table format.

‎collections/metabolomics/v2/proposals/skills/metabolomicshub-announcement-authoring/SKILL.md‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
---
2+
name: metabolomicshub-announcement-authoring
3+
description: Use when publishing a metabolomics dataset into the MetabolomicsHub index
4+
— building the common-model file, deriving and validating the announcement against
5+
the right profile, minting an identifier, and submitting a revision whose outcome
6+
only a polled task reports.
7+
license: CC-BY-4.0
8+
status: hold
9+
metadata:
10+
tools:
11+
- MetabolomicsHub
12+
- mhd-cli
13+
techniques:
14+
- LC-MS
15+
- GC-MS
16+
repo_url: https://github.com/MetabolomicsHub/mhd-model
17+
service_url: https://www.metabolomicshub.org/api/submission
18+
related_skills:
19+
- metabolomicshub-cross-repository-dataset-search
20+
license_tier: open
21+
provenance_tier: repository
22+
tool_license:
23+
tier: open
24+
requires_ack: false
25+
ref: Apache-2.0
26+
url: https://github.com/MetabolomicsHub/mhd-model/blob/main/LICENSE
27+
verified_against:
28+
package: mhd-model
29+
api_version: v0.0.1
30+
observed: '2026-08-21'
31+
schema_version: 0.2.0
32+
---
33+
34+
# metabolomicshub-announcement-authoring
35+
36+
Turn a dataset's metadata into an MHD announcement that the hub will accept, and
37+
find out that it will not before submitting rather than after.
38+
39+
## When this applies
40+
41+
A repository, a consortium partner or a submitter with a large deposit needs a
42+
study to appear in the MetabolomicsHub index. The route is not a web form: it is
43+
a common-model file, an announcement derived from it, a validation pass, an
44+
identifier, and a submitted revision. Each of those has a failure mode of its
45+
own, and two of them are asynchronous.
46+
47+
This is the depositor's side of the hub. Reading the index is a different job.
48+
49+
## The two artefacts, in the order they exist
50+
51+
1. **The MHD common data model file** — the full description of the study:
52+
samples, assays, protocols, parameters, instruments, publications.
53+
2. **The announcement file** — derived from the model file, and the thing the
54+
hub actually ingests. It references the model file by URL.
55+
56+
Deriving the second from the first is a command, not an editing task. Writing an
57+
announcement by hand and keeping it consistent with a model file is the mistake
58+
this ordering exists to prevent.
59+
60+
## Procedure
61+
62+
1. **Choose the profile before writing anything.** Two model versions (v0.1,
63+
v1.0) each carry two profiles: `ms-profile` and `legacy-profile`. The MS
64+
profile expects structured acquisition metadata; the legacy profile is the
65+
reduced shape for studies migrated from an existing repository. Both the
66+
announcement file and the common-model file have their own schema per
67+
profile, so there are four schemas in play and picking the wrong one produces
68+
validation errors that read as missing data.
69+
`GET /v0_1/schemas` returns the profiles the server currently enforces, and
70+
`GET /v0_1/server-info` reports which is the default. Ask the server; the
71+
defaults move with releases.
72+
73+
2. **Validate the model file first.**
74+
75+
```
76+
mhd-cli validate mhd <mhd_study_id> <mhd_model_file_path>
77+
```
78+
79+
Fixing the model file after deriving an announcement means deriving again.
80+
81+
3. **Derive the announcement.**
82+
83+
```
84+
mhd-cli create announcement [--output-dir DIR] [--output-filename NAME] \
85+
<mhd_study_id> <mhd_model_file_path> <target_mhd_model_file_url>
86+
```
87+
88+
The third argument is where the model file *will be* publicly readable. The
89+
hub checks that URL for accessibility when the announcement is shared, so a
90+
placeholder or a private link fails at submission rather than at derivation.
91+
Publish the model file before you submit, not before you derive.
92+
93+
4. **Validate the announcement.**
94+
95+
```
96+
mhd-cli validate announcement [--output-path FILE] <mhd_study_id> \
97+
<announcement_file_path>
98+
```
99+
100+
Use `--output-path` in any automated pipeline: it writes the result where a
101+
later step can read it, instead of leaving it in a log.
102+
103+
5. **Mint an identifier in the test lane first.** `POST /v0_1/identifiers`,
104+
or `MhdClient.get_new_mhd_accession(dataset_repository_identifier, accession_type)`.
105+
The accepted accession types are `mhd`, `legacy`, `test-mhd`, `test-legacy`
106+
and `dev`. Do a full dry run on a test accession before requesting a real
107+
one; identifiers are the part of this process you cannot take back.
108+
109+
6. **Submit the revision, then poll.** `POST /v0_1/datasets/{accession}/announcements`
110+
is a multipart upload authenticated with an `x-api-token` header, and it
111+
returns a `taskId` rather than a verdict. The outcome is read from
112+
`GET /v0_1/datasets/{accession}/tasks/{task_id}`. The shipped client polls it
113+
for you (ten attempts, five seconds apart); anything you write yourself must
114+
poll too. A 200 on the upload means the file was accepted for validation, not
115+
that the dataset was announced.
116+
117+
7. **Record the revision.** Announcements are versioned: every submission takes
118+
an `announcement_reason` and `GET /v0_1/datasets/{accession}/announcements`
119+
lists the revisions. The reason string is the only human-readable account of
120+
why a dataset's public record changed, so write it for someone reading it in
121+
two years.
122+
123+
## Related outputs from the same model file
124+
125+
`mhd-cli create sdrf` and `mhd-cli create neo4j-input` emit an SDRF table and a
126+
graph-import form of the same study. Both take the model file, so they stay
127+
consistent with the announcement by construction — worth preferring over
128+
exporting from the announcement or from a repository's own metadata.
129+
130+
## Verification
131+
132+
- The model file validates under the same profile the announcement declares.
133+
- `target_mhd_model_file_url` resolves publicly and returns the model file that
134+
was used to derive the announcement, not a later edit of it.
135+
- The submission task reached a terminal state and was read; the upload's own
136+
status code was not treated as the result.
137+
- The study appears in the search index under the expected repository, and the
138+
fields you care about are populated rather than merely present.
139+
140+
## Limitations
141+
142+
- The announcement carries what the model file carried. Fields the source
143+
repository never recorded stay empty, and the hub cannot infer them; this is
144+
the origin of the uneven field coverage seen from the search side.
145+
- Identifier minting and submission need an API token issued to a repository.
146+
A submitter without one goes through their repository, not directly.
147+
- The client and the schemas move together. Pin the `mhd-model` version used for
148+
a deposit, and re-validate rather than assuming an older announcement still
149+
conforms.
150+
- The accession types accepted by the client are the five listed above. The
151+
package README's example passes `test`, which is not among them.

0 commit comments

Comments
 (0)