Skip to content

Commit aaefc2c

Browse files
ghar1821ghar1821
authored
Add cytonorm to batch (#92)
* renamed existing cytonorm methods * added cytonorm to goal * change batch label to character to avoid confusion * Update CHANGELOG.md
1 parent fc83bb5 commit aaefc2c

15 files changed

Lines changed: 791 additions & 48 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,9 +53,12 @@
5353
* Added CytoNorm with aggregate of samples as controls (`methods/cytonorm_no_controls`).
5454
* Added parameters to tune CytoNorm.
5555

56+
57+
* Added CytoNorm correction to a goal batch (PR #92).
5658
* Added cyCombine correction to a reference batch (PR #90).
5759
* Added `metrics/bras`
5860

61+
5962
## MAJOR CHANGES
6063

6164
* Updated file schema (PR #18):

src/methods/cytonorm_all_controls/config.vsh.yaml renamed to src/methods/cytonorm_all_controls_to_goal/config.vsh.yaml

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -7,22 +7,24 @@ __merge__: ../../api/comp_method.yaml
77

88
# A unique identifier for your component (required).
99
# Can contain only lowercase letters or underscores.
10-
name: cytonorm_all_controls
10+
name: cytonorm_all_controls_to_goal
1111
# A relatively short label, used when rendering visualisations (required)
12-
label: CytoNorm with all controls
12+
label: CytoNorm (all-controls, to-goal)
1313
# A one sentence summary of how this method works (required). Used when
1414
# rendering summary tables.
15-
summary: CytoNorm with all control samples.
15+
summary: CytoNorm run with all control samples, correcting to a goal batch.
1616
# A multi-line description of how this component works (required). Used
1717
# when rendering reference documentation.
1818
description: |
1919
CytoNorm corrects batch effects by using reference control samples (aliquots of one sample,
2020
technical replicates) included with each batch.
21-
It clusters cells, then trains a model on the control samples to learn how marker
21+
It clusters cells using FlowSOM, then trains a model on the control samples to learn how marker
2222
expression distributions differ across batches for each population.
23-
It then uses splines to align these distributions to a common reference (either the mean
24-
of batches or to a single batch).
25-
Here, we run CytoNorm using all control samples available, aligning to the mean of the batches.
23+
It then uses splines to align these distributions to a common reference (either a midpoint
24+
derived from all batches or to a single batch).
25+
26+
Here, we run CytoNorm using all control samples available, aligning the batches to batch 1.
27+
2628
The parameter nQ, which specifies the number of quantiles used when computing the splines
2729
is varied linearly between value of 80-120, with default set to 99 following the default value provided by CytoNorm.
2830
Clustering was performed by FlowSOM.

‎src/methods/cytonorm_all_controls_to_goal/script.R‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
requireNamespace("flowCore", quietly = TRUE)
2+
requireNamespace("anndata", quietly = TRUE)
3+
requireNamespace("Biobase", quietly = TRUE)
4+
requireNamespace("CytoNorm", quietly = TRUE)
5+
6+
## VIASH START
7+
par <- list(
8+
input = "resources_test/task_cyto_batch_integration/cyto_spleen_subset/unintegrated_censored.h5ad",
9+
output = "resources_test/output.h5ad",
10+
som_grid_size = 10,
11+
num_metacluster = 10,
12+
n_quantiles = 99
13+
)
14+
meta <- list(
15+
name = "cytonorm_control",
16+
temp_dir = "resources_test/task_cyto_batch_integration/tmp",
17+
resources_dir = "src/utils"
18+
)
19+
## VIASH END
20+
21+
source(paste0(meta$resources_dir, "/anndata_to_fcs.R"))
22+
23+
tmp_path <- meta[["temp_dir"]]
24+
25+
cat("Reading input files\n")
26+
adata <- anndata::read_h5ad(par[["input"]])
27+
28+
cat("Preparing training data\n")
29+
30+
# get the control samples to be used for training the model
31+
fset_train <- anndata_to_fcs(adata[adata$obs$is_control != 0, ])
32+
# every sample, including the controls, pretty much the entire unintegrated data
33+
# will be corrected.
34+
fset_all <- anndata_to_fcs(adata)
35+
36+
cat("Setting up some variables for training the model\n")
37+
38+
# get batch label for the training data
39+
batch_lab_train <- vapply(sampleNames(fset_train), function(samp) {
40+
as.character(
41+
unique(
42+
adata[adata$obs$sample == samp]$obs$batch
43+
)[1]
44+
)
45+
}, FUN.VALUE = character(1))
46+
47+
# get batch label for the all data
48+
batch_labs <- vapply(sampleNames(fset_all), function(samp) {
49+
as.character(
50+
unique(
51+
adata[adata$obs$sample == samp]$obs$batch
52+
)[1]
53+
)
54+
}, FUN.VALUE = character(1))
55+
56+
markers_to_correct <- as.vector(adata$var$channel[adata$var$to_correct])
57+
58+
lineage_markers <- as.vector(adata$var$channel[adata$var$marker_type == "lineage"])
59+
60+
# get number of cells for clustering.
61+
# we will define this as the minimum of the smallest sample and 1,000,000.
62+
# and multiply this by how many samples we have - because internally,
63+
# this number is divided by the number of files to determine the amount to select from
64+
# each individual file.
65+
n_cells_per_control_sample <- flowCore::fsApply(fset_train, function(ff) nrow(exprs(ff)))
66+
n_cells_for_clustering <- min(n_cells_per_control_sample, 1000000) * length(n_cells_per_control_sample)
67+
68+
cat("Training Cytonorm model using all control samples\n")
69+
70+
# FlowSOM.params and normParams are the default parameters in cytonorm
71+
model <- CytoNorm::CytoNorm.train(
72+
files = fset_train,
73+
labels = batch_lab_train,
74+
channels = markers_to_correct,
75+
outputDir = tmp_path,
76+
FlowSOM.params = list(
77+
nCells = n_cells_for_clustering,
78+
xdim = par[["som_grid_size"]],
79+
ydim = par[["som_grid_size"]],
80+
nClus = par[["num_metacluster"]],
81+
scale = FALSE,
82+
colsToUse = lineage_markers
83+
),
84+
transformList = NULL,
85+
normParams = list(
86+
nQ = par[["n_quantiles"]],
87+
goal = "1"
88+
),
89+
seed = 42,
90+
verbose = FALSE,
91+
recompute = TRUE
92+
)
93+
94+
cat("Normalising using Cytonorm model trained using all control samples\n")
95+
96+
norm_fset_all <- CytoNorm::CytoNorm.normalize(
97+
model = model,
98+
files = fset_all,
99+
labels = batch_labs,
100+
transformList = NULL,
101+
transformList.reverse = NULL,
102+
outputDir = tmp_path,
103+
prefix = "Norm_",
104+
clean = TRUE,
105+
write = FALSE,
106+
verbose = FALSE
107+
)
108+
109+
cat("Preparing output anndata\n")
110+
# cytonorm will return all markers corrected or not in the same order as the input data.
111+
# so we can just directly replace the colnames with var_names
112+
norm_mat <- flowCore::fsApply(norm_fset_all, exprs)
113+
colnames(norm_mat) <- adata$var_names
114+
115+
norm_mat <- anndata::AnnData(
116+
obs = adata$obs[, integer(0)],
117+
var = adata$var[colnames(norm_mat), integer(0)],
118+
layers = list(integrated = norm_mat),
119+
uns = list(
120+
dataset_id = adata$uns$dataset_id,
121+
method_id = meta$name,
122+
parameters = list()
123+
)
124+
)
125+
126+
cat("Write output AnnData to file\n")
127+
norm_mat$write_h5ad(par[["output"]], compression = "gzip")

‎src/methods/cytonorm_all_controls_to_mid/config.vsh.yaml‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# The API specifies which type of component this is.
2+
# It contains specifications for:
3+
# - The input/output files
4+
# - Common parameters
5+
# - A unit test
6+
__merge__: ../../api/comp_method.yaml
7+
8+
# A unique identifier for your component (required).
9+
# Can contain only lowercase letters or underscores.
10+
name: cytonorm_all_controls_to_mid
11+
# A relatively short label, used when rendering visualisations (required)
12+
label: CytoNorm (all-controls, to-middle)
13+
# A one sentence summary of how this method works (required). Used when
14+
# rendering summary tables.
15+
summary: CytoNorm run with all control samples, correcting to a midpoint.
16+
# A multi-line description of how this component works (required). Used
17+
# when rendering reference documentation.
18+
description: |
19+
CytoNorm corrects batch effects by using reference control samples (aliquots of one sample,
20+
technical replicates) included with each batch.
21+
It clusters cells using FlowSOM, then trains a model on the control samples to learn how marker
22+
expression distributions differ across batches for each population.
23+
It then uses splines to align these distributions to a common reference (either a midpoint
24+
derived from all batches or to a single batch).
25+
26+
Here, we run CytoNorm using all control samples available, aligning to a midpoint derived from all batches.
27+
28+
The parameter nQ, which specifies the number of quantiles used when computing the splines
29+
is varied linearly between value of 80-120, with default set to 99 following the default value provided by CytoNorm.
30+
Clustering was performed by FlowSOM.
31+
The number of cells clustered by FlowSOM is set to be number of cells in the smallest
32+
control samples or 1,000,000, whichever is the smaller, multiplied by how many control samples
33+
there are in the data.
34+
The size of the SOM grid is varied linearly between value of 6-16, with default set to 15
35+
following the default value provided by CytoNorm.
36+
The number of metaclusters is varied linearly between value of 8-20, with default set to 10
37+
following the default value provided by CytoNorm.
38+
39+
references:
40+
doi:
41+
- 10.1002/cyto.a.23904
42+
links:
43+
# URL to the documentation for this method (required).
44+
documentation: https://github.com/saeyslab/CytoNorm
45+
# URL to the code repository for this method (required).
46+
repository: https://github.com/saeyslab/CytoNorm
47+
48+
argument_groups:
49+
- name: Parameters
50+
arguments:
51+
- type: integer
52+
name: --som_grid_size
53+
info:
54+
optimize:
55+
type: linear
56+
lower: 6
57+
upper: 16
58+
default: 15
59+
description: SOM grid size used when training CytoNorm model.
60+
- type: integer
61+
name: --num_metacluster
62+
info:
63+
optimize:
64+
type: linear
65+
lower: 8
66+
upper: 20
67+
default: 10
68+
description: Number of metaclusters generated when training CytoNorm model.
69+
- type: integer
70+
name: --n_quantiles
71+
info:
72+
optimize:
73+
type: linear
74+
lower: 80
75+
upper: 120
76+
default: 99
77+
description: Number of quantiles to use when training the CytoNorm model.
78+
79+
# Metadata for your component
80+
# Resources required to run the component
81+
resources:
82+
# The script of your component (required)
83+
- type: r_script
84+
path: script.R
85+
- path: /src/utils/anndata_to_fcs.R
86+
87+
engines:
88+
# Specifications for the Docker image for this component.
89+
- type: docker
90+
image: openproblems/base_r:1
91+
# Add custom dependencies here (optional). For more information, see
92+
# https://viash.io/reference/config/engines/docker/#setup .
93+
setup:
94+
- type: r
95+
github: [saeyslab/cytoNorm]
96+
bioc: [ flowCore, Biobase ]
97+
packages: [ anndata, docstring ]
98+
99+
runners:
100+
# This platform allows running the component natively
101+
- type: executable
102+
# Allows turning the component into a Nextflow module / pipeline.
103+
- type: nextflow
104+
directives:
105+
label: [midtime,midmem,midcpu]

src/methods/cytonorm_all_controls/script.R renamed to src/methods/cytonorm_all_controls_to_mid/script.R

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -36,9 +36,22 @@ fset_all <- anndata_to_fcs(adata)
3636
cat("Setting up some variables for training the model\n")
3737

3838
# get batch label for the training data
39-
batch_lab_train <- sapply(sampleNames(fset_train), function(samp) {
40-
unique(adata[adata$obs$sample == samp]$obs$batch)[1]
41-
})
39+
batch_lab_train <- vapply(sampleNames(fset_train), function(samp) {
40+
as.character(
41+
unique(
42+
adata[adata$obs$sample == samp]$obs$batch
43+
)[1]
44+
)
45+
}, FUN.VALUE = character(1))
46+
47+
# get batch label for the all data
48+
batch_labs <- vapply(sampleNames(fset_all), function(samp) {
49+
as.character(
50+
unique(
51+
adata[adata$obs$sample == samp]$obs$batch
52+
)[1]
53+
)
54+
}, FUN.VALUE = character(1))
4255

4356
markers_to_correct <- as.vector(adata$var$channel[adata$var$to_correct])
4457

@@ -74,11 +87,6 @@ model <- CytoNorm::CytoNorm.train(
7487
verbose = FALSE
7588
)
7689

77-
# get batch label for the validation data
78-
batch_labs <- sapply(sampleNames(fset_all), function(samp) {
79-
unique(adata[adata$obs$sample == samp]$obs$batch)[1]
80-
})
81-
8290
cat("Normalising using Cytonorm model trained using all control samples\n")
8391

8492
norm_fset_all <- CytoNorm::CytoNorm.normalize(

src/methods/cytonorm_no_controls/config.vsh.yaml renamed to src/methods/cytonorm_no_controls_to_goal/config.vsh.yaml

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -7,23 +7,25 @@ __merge__: ../../api/comp_method.yaml
77

88
# A unique identifier for your component (required).
99
# Can contain only lowercase letters or underscores.
10-
name: cytonorm_no_controls
10+
name: cytonorm_no_controls_to_goal
1111
# A relatively short label, used when rendering visualisations (required)
12-
label: CytoNorm without controls
12+
label: CytoNorm (no-controls, to-goal)
1313
# A one sentence summary of how this method works (required). Used when
1414
# rendering summary tables.
15-
summary: CytoNorm without control samples.
15+
summary: CytoNorm run without control samples, correcting to a goal batch.
1616
# A multi-line description of how this component works (required). Used
1717
# when rendering reference documentation.
1818
description: |
1919
CytoNorm corrects batch effects by using reference control samples (aliquots of one sample,
2020
technical replicates) included with each batch.
21-
It clusters cells, then trains a model on the control samples to learn how marker
21+
It clusters cells using FlowSOM, then trains a model on the control samples to learn how marker
2222
expression distributions differ across batches for each population.
23-
It then uses splines to align these distributions to a common reference (either the mean
24-
of batches or to a single batch).
23+
It then uses splines to align these distributions to a common reference (either a midpoint
24+
derived from all batches or to a single batch).
25+
2526
In this CytoNorm version, an aggregate of each batch is created and subsequently used as a
26-
proxy for the control samples.
27+
proxy for the control samples, aligning the batches to batch 1.
28+
2729
The number of cells used to create an aggregate is set as the number of cells in the smallest
2830
sample or 1,000,000, whichever is the smaller, multiplied by how many samples there are in the batch.
2931
Clustering was performed by FlowSOM.

0 commit comments

Comments
 (0)