Skip to content

Commit df3cfd3

Browse files
rcannoodrcannood
committed
adapt api files
1 parent aaefc2c commit df3cfd3

6 files changed

Lines changed: 113 additions & 96 deletions

File tree

‎README.md‎

Lines changed: 93 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -46,27 +46,27 @@ signals were preserved.
4646
flowchart TB
4747
file_common_dataset("<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#file-format-common-dataset'>Common Dataset</a>")
4848
comp_data_processor[/"<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#component-type-data-processor'>Data processor</a>"/]
49-
file_unintegrated_censored("<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#file-format-unintegrated-censored'>Unintegrated Censored</a>")
49+
file_censored("<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#file-format-unintegrated-censored'>Unintegrated Censored</a>")
5050
file_unintegrated("<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#file-format-unintegrated'>Unintegrated</a>")
51-
file_validation("<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#file-format-validation'>Validation</a>")
5251
comp_method[/"<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#component-type-method'>Method</a>"/]
5352
comp_control_method[/"<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#component-type-control-method'>Control Method</a>"/]
5453
comp_metric[/"<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#component-type-metric'>Metric</a>"/]
5554
file_integrated("<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#file-format-integrated'>Integrated</a>")
5655
file_score("<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#file-format-score'>Score</a>")
56+
file_validation("<a href='https://github.com/openproblems-bio/task_cyto_batch_integration#file-format-validation'>Validation</a>")
5757
file_common_dataset---comp_data_processor
58-
comp_data_processor-->file_unintegrated_censored
58+
comp_data_processor-->file_censored
59+
comp_data_processor-->file_censored
5960
comp_data_processor-->file_unintegrated
60-
comp_data_processor-->file_validation
61-
file_unintegrated_censored---comp_method
61+
file_censored---comp_method
6262
file_unintegrated---comp_control_method
6363
file_unintegrated---comp_metric
64-
file_validation---comp_control_method
65-
file_validation---comp_metric
6664
comp_method-->file_integrated
6765
comp_control_method-->file_integrated
6866
comp_metric-->file_score
6967
file_integrated---comp_metric
68+
file_integrated---comp_metric
69+
file_validation---comp_control_method
7070
```
7171

7272
## File format: Common Dataset
@@ -131,24 +131,30 @@ Arguments:
131131
| Name | Type | Description |
132132
|:---|:---|:---|
133133
| `--input` | `file` | A subset of the common dataset. |
134-
| `--output_unintegrated_censored` | `file` | (*Output*) An unintegrated dataset with certain columns (cells metadata), such as the donor information, hidden. These columns are intentionally hidden to prevent bias. The batch correction algorithm should not have to rely on these information to properly integrate different batches. This dataset is used as the input for the batch correction algorithm. The cells therein are identical to those in the unintegrated dataset. |
135-
| `--output_unintegrated` | `file` | (*Output*) The complete unintegrated dataset, including all cells’ metadata (columns) from the unintegrated_censored dataset. The cells in this dataset are the same to those in the unintegrated_censored dataset. |
136-
| `--output_validation` | `file` | (*Output*) Hold-out dataset for validation. |
134+
| `--output_censored_left` | `file` | (*Output*) An unintegrated dataset with certain columns (cells metadata), such as the donor information, hidden. These columns are intentionally hidden to prevent bias. |
135+
| `--output_censored_right` | `file` | (*Output*) An unintegrated dataset with certain columns (cells metadata), such as the donor information, hidden. These columns are intentionally hidden to prevent bias. |
136+
| `--output_unintegrated` | `file` | (*Output*) The complete unintegrated dataset, including all cells’ metadata (columns) from the unintegrated_censored dataset. |
137137

138138
</div>
139139

140140
## File format: Unintegrated Censored
141141

142+
An unintegrated dataset with certain columns (cells metadata), such as
143+
the donor information, hidden. These columns are intentionally hidden to
144+
prevent bias.
145+
146+
Example file:
147+
`resources_test/task_cyto_batch_integration/mouse_spleen_flow_cytometry_subset/unintegrated_censored.h5ad`
148+
149+
Description:
150+
142151
An unintegrated dataset with certain columns (cells metadata), such as
143152
the donor information, hidden. These columns are intentionally hidden to
144153
prevent bias. The batch correction algorithm should not have to rely on
145154
these information to properly integrate different batches. This dataset
146155
is used as the input for the batch correction algorithm. The cells
147156
therein are identical to those in the unintegrated dataset.
148157

149-
Example file:
150-
`resources_test/task_cyto_batch_integration/mouse_spleen_flow_cytometry_subset/unintegrated_censored.h5ad`
151-
152158
Format:
153159

154160
<div class="small">
@@ -189,76 +195,16 @@ Data structure:
189195
## File format: Unintegrated
190196

191197
The complete unintegrated dataset, including all cells’ metadata
192-
(columns) from the unintegrated_censored dataset. The cells in this
193-
dataset are the same to those in the unintegrated_censored dataset.
198+
(columns) from the unintegrated_censored dataset.
194199

195200
Example file:
196201
`resources_test/task_cyto_batch_integration/mouse_spleen_flow_cytometry_subset/unintegrated.h5ad`
197202

198-
Format:
199-
200-
<div class="small">
201-
202-
AnnData object
203-
obs: 'cell_type', 'batch', 'sample', 'donor', 'group', 'is_control'
204-
var: 'numeric_id', 'channel', 'marker', 'marker_type', 'to_correct'
205-
layers: 'preprocessed'
206-
uns: 'dataset_id', 'dataset_name', 'dataset_url', 'dataset_reference', 'dataset_summary', 'dataset_description', 'dataset_organism', 'parameter_som_xdim', 'parameter_som_ydim', 'parameter_num_clusters'
207-
208-
</div>
209-
210-
Data structure:
211-
212-
<div class="small">
213-
214-
| Slot | Type | Description |
215-
|:---|:---|:---|
216-
| `obs["cell_type"]` | `string` | Cell type information. |
217-
| `obs["batch"]` | `string` | Batch information. |
218-
| `obs["sample"]` | `string` | Sample ID. |
219-
| `obs["donor"]` | `string` | Donor ID. |
220-
| `obs["group"]` | `string` | Biological group of the donor. |
221-
| `obs["is_control"]` | `integer` | Whether the sample the cell came from can be used as a control for batch effect correction. 0: cannot be used as a control. \>= 1: can be used as a control. For cells with \>= 1: cells with the same value come from the same donor. Different values indicate different donors. |
222-
| `var["numeric_id"]` | `integer` | Numeric ID associated with each marker. |
223-
| `var["channel"]` | `string` | The channel / detector of the instrument. |
224-
| `var["marker"]` | `string` | (*Optional*) The marker name associated with the channel. |
225-
| `var["marker_type"]` | `string` | Whether the marker is a functional or lineage marker. |
226-
| `var["to_correct"]` | `boolean` | Whether the marker will be batch corrected. |
227-
| `layers["preprocessed"]` | `double` | preprocessed data, e.g. already compensated, transformed and debris/doublets removed. |
228-
| `uns["dataset_id"]` | `string` | A unique identifier for the dataset. |
229-
| `uns["dataset_name"]` | `string` | Nicely formatted name. |
230-
| `uns["dataset_url"]` | `string` | (*Optional*) Link to the original source of the dataset. |
231-
| `uns["dataset_reference"]` | `string` | (*Optional*) Bibtex reference of the paper in which the dataset was published. |
232-
| `uns["dataset_summary"]` | `string` | Short description of the dataset. |
233-
| `uns["dataset_description"]` | `string` | Long description of the dataset. |
234-
| `uns["dataset_organism"]` | `string` | (*Optional*) The organism of the sample in the dataset. |
235-
| `uns["parameter_som_xdim"]` | `integer` | Parameter used to define the dimensions of the self-organizing map (SOM) grid used for flowsom clustering. |
236-
| `uns["parameter_som_ydim"]` | `integer` | Parameter used to define the dimensions of the self-organizing map (SOM) grid used for flowsom clustering. |
237-
| `uns["parameter_num_clusters"]` | `integer` | Parameter used to define the number of clusters used in the meta-clustering step of the flowsom algorithm. |
238-
239-
</div>
240-
241-
## File format: Validation
242-
243-
Hold-out dataset for validation.
244-
245-
Example file:
246-
`resources_test/task_cyto_batch_integration/mouse_spleen_flow_cytometry_subset/validation.h5ad`
247-
248203
Description:
249204

250-
Dataset containing cells from samples that were held out for evaluating
251-
batch integration output. The cells that are in this dataset belong to
252-
samples which are not included in the unintegrated or
253-
unintegrated_censored datasets. For example, if samples from donor A are
254-
present in batch 1 and 2, the sample from batch 1 may be used as input
255-
for the batch correction algorithm (and thus present in unintegrated and
256-
unintegrated_censored datasets). The sample from batch 2, may not be
257-
included as an input for the batch correction algorithm, but is needed
258-
to validate whether whether the algorithm managed to correct the batch
259-
effect in batch 2 towards batch 1. This sample will then be included in
260-
this dataset (but not in unintegrated and unintegrated_censored
261-
datasets).
205+
The complete unintegrated dataset, including all cells’ metadata
206+
(columns) from the unintegrated_censored dataset. The cells in this
207+
dataset are the same to those in the unintegrated_censored dataset.
262208

263209
Format:
264210

@@ -313,7 +259,7 @@ Arguments:
313259

314260
| Name | Type | Description |
315261
|:---|:---|:---|
316-
| `--input` | `file` | An unintegrated dataset with certain columns (cells metadata), such as the donor information, hidden. These columns are intentionally hidden to prevent bias. The batch correction algorithm should not have to rely on these information to properly integrate different batches. This dataset is used as the input for the batch correction algorithm. The cells therein are identical to those in the unintegrated dataset. |
262+
| `--input` | `file` | An unintegrated dataset with certain columns (cells metadata), such as the donor information, hidden. These columns are intentionally hidden to prevent bias. |
317263
| `--output` | `file` | (*Output*) Integrated dataset which batch effect was corrected by an algorithm. |
318264

319265
</div>
@@ -328,7 +274,7 @@ Arguments:
328274

329275
| Name | Type | Description |
330276
|:---|:---|:---|
331-
| `--input_unintegrated` | `file` | The complete unintegrated dataset, including all cells’ metadata (columns) from the unintegrated_censored dataset. The cells in this dataset are the same to those in the unintegrated_censored dataset. |
277+
| `--input_unintegrated` | `file` | The complete unintegrated dataset, including all cells’ metadata (columns) from the unintegrated_censored dataset. |
332278
| `--input_validation` | `file` | Hold-out dataset for validation. |
333279
| `--output` | `file` | (*Output*) Integrated dataset which batch effect was corrected by an algorithm. |
334280

@@ -344,9 +290,9 @@ Arguments:
344290

345291
| Name | Type | Description |
346292
|:---|:---|:---|
347-
| `--input_validation` | `file` | Hold-out dataset for validation. |
348-
| `--input_unintegrated` | `file` | The complete unintegrated dataset, including all cells’ metadata (columns) from the unintegrated_censored dataset. The cells in this dataset are the same to those in the unintegrated_censored dataset. |
349-
| `--input_integrated` | `file` | Integrated dataset which batch effect was corrected by an algorithm. |
293+
| `--input_unintegrated` | `file` | The complete unintegrated dataset, including all cells’ metadata (columns) from the unintegrated_censored dataset. |
294+
| `--input_integrated_left` | `file` | Integrated dataset which batch effect was corrected by an algorithm. |
295+
| `--input_integrated_right` | `file` | Integrated dataset which batch effect was corrected by an algorithm. |
350296
| `--output` | `file` | (*Output*) File indicating the score of a metric. |
351297

352298
</div>
@@ -410,3 +356,68 @@ Data structure:
410356

411357
</div>
412358

359+
## File format: Validation
360+
361+
Hold-out dataset for validation.
362+
363+
Example file:
364+
`resources_test/task_cyto_batch_integration/mouse_spleen_flow_cytometry_subset/validation.h5ad`
365+
366+
Description:
367+
368+
Dataset containing cells from samples that were held out for evaluating
369+
batch integration output. The cells that are in this dataset belong to
370+
samples which are not included in the unintegrated or
371+
unintegrated_censored datasets. For example, if samples from donor A are
372+
present in batch 1 and 2, the sample from batch 1 may be used as input
373+
for the batch correction algorithm (and thus present in unintegrated and
374+
unintegrated_censored datasets). The sample from batch 2, may not be
375+
included as an input for the batch correction algorithm, but is needed
376+
to validate whether whether the algorithm managed to correct the batch
377+
effect in batch 2 towards batch 1. This sample will then be included in
378+
this dataset (but not in unintegrated and unintegrated_censored
379+
datasets).
380+
381+
Format:
382+
383+
<div class="small">
384+
385+
AnnData object
386+
obs: 'cell_type', 'batch', 'sample', 'donor', 'group', 'is_control'
387+
var: 'numeric_id', 'channel', 'marker', 'marker_type', 'to_correct'
388+
layers: 'preprocessed'
389+
uns: 'dataset_id', 'dataset_name', 'dataset_url', 'dataset_reference', 'dataset_summary', 'dataset_description', 'dataset_organism', 'parameter_som_xdim', 'parameter_som_ydim', 'parameter_num_clusters'
390+
391+
</div>
392+
393+
Data structure:
394+
395+
<div class="small">
396+
397+
| Slot | Type | Description |
398+
|:---|:---|:---|
399+
| `obs["cell_type"]` | `string` | Cell type information. |
400+
| `obs["batch"]` | `string` | Batch information. |
401+
| `obs["sample"]` | `string` | Sample ID. |
402+
| `obs["donor"]` | `string` | Donor ID. |
403+
| `obs["group"]` | `string` | Biological group of the donor. |
404+
| `obs["is_control"]` | `integer` | Whether the sample the cell came from can be used as a control for batch effect correction. 0: cannot be used as a control. \>= 1: can be used as a control. For cells with \>= 1: cells with the same value come from the same donor. Different values indicate different donors. |
405+
| `var["numeric_id"]` | `integer` | Numeric ID associated with each marker. |
406+
| `var["channel"]` | `string` | The channel / detector of the instrument. |
407+
| `var["marker"]` | `string` | (*Optional*) The marker name associated with the channel. |
408+
| `var["marker_type"]` | `string` | Whether the marker is a functional or lineage marker. |
409+
| `var["to_correct"]` | `boolean` | Whether the marker will be batch corrected. |
410+
| `layers["preprocessed"]` | `double` | preprocessed data, e.g. already compensated, transformed and debris/doublets removed. |
411+
| `uns["dataset_id"]` | `string` | A unique identifier for the dataset. |
412+
| `uns["dataset_name"]` | `string` | Nicely formatted name. |
413+
| `uns["dataset_url"]` | `string` | (*Optional*) Link to the original source of the dataset. |
414+
| `uns["dataset_reference"]` | `string` | (*Optional*) Bibtex reference of the paper in which the dataset was published. |
415+
| `uns["dataset_summary"]` | `string` | Short description of the dataset. |
416+
| `uns["dataset_description"]` | `string` | Long description of the dataset. |
417+
| `uns["dataset_organism"]` | `string` | (*Optional*) The organism of the sample in the dataset. |
418+
| `uns["parameter_som_xdim"]` | `integer` | Parameter used to define the width of the self-organizing map (SOM) grid. Usually between 10 and 20. |
419+
| `uns["parameter_som_ydim"]` | `integer` | Parameter used to define the height of the self-organizing map (SOM) grid. Usually between 10 and 20. |
420+
| `uns["parameter_num_clusters"]` | `integer` | Parameter used to define the number of clusters. Set this number to be slightly higher than the number of cell types expected in the dataset. |
421+
422+
</div>
423+

‎src/api/comp_data_processor.yaml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,16 +11,16 @@ arguments:
1111
__merge__: file_common_dataset.yaml
1212
direction: input
1313
required: true
14-
- name: "--output_unintegrated_censored"
15-
__merge__: file_unintegrated_censored.yaml
14+
- name: "--output_censored_left"
15+
__merge__: file_censored.yaml
1616
direction: output
1717
required: true
18-
- name: "--output_unintegrated"
19-
__merge__: file_unintegrated.yaml
18+
- name: "--output_censored_right"
19+
__merge__: file_censored.yaml
2020
direction: output
2121
required: true
22-
- name: "--output_validation"
23-
__merge__: file_validation.yaml
22+
- name: "--output_unintegrated"
23+
__merge__: file_unintegrated.yaml
2424
direction: output
2525
required: true
2626
test_resources:

‎src/api/comp_method.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ info:
88
A method for integrating batch effects in cytometry data.
99
arguments:
1010
- name: --input
11-
__merge__: file_unintegrated_censored.yaml
11+
__merge__: file_censored.yaml
1212
required: true
1313
direction: input
1414
- name: --output

‎src/api/comp_metric.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,15 +7,15 @@ info:
77
description: |
88
A metric for evaluating method predictions.
99
arguments:
10-
- name: "--input_validation"
11-
__merge__: file_validation.yaml
12-
direction: input
13-
required: true
1410
- name: "--input_unintegrated"
1511
__merge__: file_unintegrated.yaml
1612
direction: input
1713
required: true
18-
- name: "--input_integrated"
14+
- name: "--input_integrated_left"
15+
__merge__: file_integrated.yaml
16+
direction: input
17+
required: true
18+
- name: "--input_integrated_right"
1919
__merge__: file_integrated.yaml
2020
direction: input
2121
required: true

src/api/file_unintegrated_censored.yaml renamed to src/api/file_censored.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ label: Unintegrated Censored
55
summary: |
66
An unintegrated dataset with certain columns (cells metadata), such as the donor information, hidden.
77
These columns are intentionally hidden to prevent bias.
8+
description: |
9+
An unintegrated dataset with certain columns (cells metadata), such as the donor information, hidden.
10+
These columns are intentionally hidden to prevent bias.
811
The batch correction algorithm should not have to rely on these information
912
to properly integrate different batches.
1013
This dataset is used as the input for the batch correction algorithm.

‎src/api/file_unintegrated.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,11 @@ type: file
33
example: "resources_test/task_cyto_batch_integration/mouse_spleen_flow_cytometry_subset/unintegrated.h5ad"
44
label: Unintegrated
55
summary: |
6-
The complete unintegrated dataset, including all cells' metadata (columns) from the
7-
unintegrated_censored dataset.
6+
The complete unintegrated dataset, including all cells' metadata (columns) from the
7+
unintegrated_censored dataset.
8+
description: |
9+
The complete unintegrated dataset, including all cells' metadata (columns) from the
10+
unintegrated_censored dataset.
811
The cells in this dataset are the same to those in the unintegrated_censored dataset.
912
info:
1013
format:

0 commit comments

Comments
 (0)