Add cytonorm to batch

CHANGELOG.md

@@ -53,9 +53,12 @@
   * Added CytoNorm with aggregate of samples as controls (`methods/cytonorm_no_controls`).
   * Added parameters to tune CytoNorm.
 
+
+* Added CytoNorm correction to a goal batch (PR #92).
 * Added cyCombine correction to a reference batch (PR #90).
 * Added `metrics/bras`
 
+
 ## MAJOR CHANGES
 
 * Updated file schema (PR #18): 

src/methods/cytonorm_all_controls/config.vsh.yaml → src/methods/cytonorm_all_controls_to_goal/config.vsh.yaml

@@ -7,22 +7,24 @@ __merge__: ../../api/comp_method.yaml
 
 # A unique identifier for your component (required).
 # Can contain only lowercase letters or underscores.
-name: cytonorm_all_controls
+name: cytonorm_all_controls_to_goal
 # A relatively short label, used when rendering visualisations (required)
-label: CytoNorm with all controls
+label: CytoNorm (all-controls, to-goal)
 # A one sentence summary of how this method works (required). Used when 
 # rendering summary tables.
-summary: CytoNorm with all control samples.
+summary: CytoNorm run with all control samples, correcting to a goal batch.
 # A multi-line description of how this component works (required). Used
 # when rendering reference documentation.
 description: |
   CytoNorm corrects batch effects by using reference control samples (aliquots of one sample, 
   technical replicates) included with each batch. 
-  It clusters cells, then trains a model on the control samples to learn how marker 
+  It clusters cells using FlowSOM, then trains a model on the control samples to learn how marker 
   expression distributions differ across batches for each population.
-  It then uses splines to align these distributions to a common reference (either the mean
-  of batches or to a single batch).
-  Here, we run CytoNorm using all control samples available, aligning to the mean of the batches.
+  It then uses splines to align these distributions to a common reference (either a midpoint
+  derived from all batches or to a single batch).
+
+  Here, we run CytoNorm using all control samples available, aligning the batches to batch 1.
+
   The parameter nQ, which specifies the number of quantiles used when computing the splines
   is varied linearly between value of 80-120, with default set to 99 following the default value provided by CytoNorm.
   Clustering was performed by FlowSOM.

src/methods/cytonorm_all_controls_to_goal/script.R

@@ -0,0 +1,127 @@
+requireNamespace("flowCore", quietly = TRUE)
+requireNamespace("anndata", quietly = TRUE)
+requireNamespace("Biobase", quietly = TRUE)
+requireNamespace("CytoNorm", quietly = TRUE)
+
+## VIASH START
+par <- list(
+    input = "resources_test/task_cyto_batch_integration/cyto_spleen_subset/unintegrated_censored.h5ad",
+    output = "resources_test/output.h5ad",
+    som_grid_size = 10,
+    num_metacluster = 10,
+    n_quantiles = 99
+)
+meta <- list(
+    name = "cytonorm_control",
+    temp_dir = "resources_test/task_cyto_batch_integration/tmp",
+    resources_dir = "src/utils"
+)
+## VIASH END
+
+source(paste0(meta$resources_dir, "/anndata_to_fcs.R"))
+
+tmp_path <- meta[["temp_dir"]]
+
+cat("Reading input files\n")
+adata <- anndata::read_h5ad(par[["input"]])
+
+cat("Preparing training data\n")
+
+# get the control samples to be used for training the model
+fset_train <- anndata_to_fcs(adata[adata$obs$is_control != 0, ])
+# every sample, including the controls, pretty much the entire unintegrated data
+# will be corrected.
+fset_all <- anndata_to_fcs(adata)
+
+cat("Setting up some variables for training the model\n")
+
+# get batch label for the training data
+batch_lab_train <- vapply(sampleNames(fset_train), function(samp) {
+    as.character(
+        unique(
+            adata[adata$obs$sample == samp]$obs$batch
+        )[1]
+    )
+}, FUN.VALUE = character(1))
+
+# get batch label for the all data
+batch_labs <- vapply(sampleNames(fset_all), function(samp) {
+    as.character(
+        unique(
+            adata[adata$obs$sample == samp]$obs$batch
+        )[1]
+    )
+}, FUN.VALUE = character(1))
+
+markers_to_correct <- as.vector(adata$var$channel[adata$var$to_correct])
+
+lineage_markers <- as.vector(adata$var$channel[adata$var$marker_type == "lineage"])
+
+# get number of cells for clustering.
+# we will define this as the minimum of the smallest sample and 1,000,000.
+# and multiply this by how many samples we have - because internally,
+# this number is divided by the number of files to determine the amount to select from
+# each individual file.
+n_cells_per_control_sample <- flowCore::fsApply(fset_train, function(ff) nrow(exprs(ff)))
+n_cells_for_clustering <- min(n_cells_per_control_sample, 1000000) * length(n_cells_per_control_sample)
+
+cat("Training Cytonorm model using all control samples\n")
+
+# FlowSOM.params and normParams are the default parameters in cytonorm
+model <- CytoNorm::CytoNorm.train(
+    files = fset_train,
+    labels = batch_lab_train,
+    channels = markers_to_correct,
+    outputDir = tmp_path,
+    FlowSOM.params = list(
+        nCells = n_cells_for_clustering,
+        xdim = par[["som_grid_size"]],
+        ydim = par[["som_grid_size"]],
+        nClus = par[["num_metacluster"]],
+        scale = FALSE,
+        colsToUse = lineage_markers
+    ),
+    transformList = NULL,
+    normParams = list(
+        nQ = par[["n_quantiles"]],
+        goal = "1"
+    ),
+    seed = 42,
+    verbose = FALSE,
+    recompute = TRUE
+)
+
+cat("Normalising using Cytonorm model trained using all control samples\n")
+
+norm_fset_all <- CytoNorm::CytoNorm.normalize(
+    model = model,
+    files = fset_all,
+    labels = batch_labs,
+    transformList = NULL,
+    transformList.reverse = NULL,
+    outputDir = tmp_path,
+    prefix = "Norm_",
+    clean = TRUE,
+    write = FALSE,
+    verbose = FALSE
+)
+
+cat("Preparing output anndata\n")
+# cytonorm will return all markers corrected or not in the same order as the input data.
+# so we can just directly replace the colnames with var_names
+norm_mat <- flowCore::fsApply(norm_fset_all, exprs)
+colnames(norm_mat) <- adata$var_names
+
+norm_mat <- anndata::AnnData(
+    obs = adata$obs[, integer(0)],
+    var = adata$var[colnames(norm_mat), integer(0)],
+    layers = list(integrated = norm_mat),
+    uns = list(
+        dataset_id = adata$uns$dataset_id,
+        method_id = meta$name,
+        parameters = list()
+    )
+)
+
+cat("Write output AnnData to file\n")
+norm_mat$write_h5ad(par[["output"]], compression = "gzip")

src/methods/cytonorm_all_controls_to_mid/config.vsh.yaml

@@ -0,0 +1,105 @@
+# The API specifies which type of component this is.
+# It contains specifications for:
+#   - The input/output files
+#   - Common parameters
+#   - A unit test
+__merge__: ../../api/comp_method.yaml
+
+# A unique identifier for your component (required).
+# Can contain only lowercase letters or underscores.
+name: cytonorm_all_controls_to_mid
+# A relatively short label, used when rendering visualisations (required)
+label: CytoNorm (all-controls, to-middle)
+# A one sentence summary of how this method works (required). Used when 
+# rendering summary tables.
+summary: CytoNorm run with all control samples, correcting to a midpoint.
+# A multi-line description of how this component works (required). Used
+# when rendering reference documentation.
+description: |
+  CytoNorm corrects batch effects by using reference control samples (aliquots of one sample, 
+  technical replicates) included with each batch. 
+  It clusters cells using FlowSOM, then trains a model on the control samples to learn how marker 
+  expression distributions differ across batches for each population.
+  It then uses splines to align these distributions to a common reference (either a midpoint
+  derived from all batches or to a single batch).
+
+  Here, we run CytoNorm using all control samples available, aligning to a midpoint derived from all batches.
+
+  The parameter nQ, which specifies the number of quantiles used when computing the splines
+  is varied linearly between value of 80-120, with default set to 99 following the default value provided by CytoNorm.
+  Clustering was performed by FlowSOM.
+  The number of cells clustered by FlowSOM is set to be number of cells in the smallest
+  control samples or 1,000,000, whichever is the smaller, multiplied by how many control samples
+  there are in the data.
+  The size of the SOM grid is varied linearly between value of 6-16, with default set to 15
+  following the default value provided by CytoNorm.
+  The number of metaclusters is varied linearly between value of 8-20, with default set to 10
+  following the default value provided by CytoNorm.
+
+references:
+  doi: 
+    - 10.1002/cyto.a.23904
+links:
+  # URL to the documentation for this method (required).
+  documentation: https://github.com/saeyslab/CytoNorm
+  # URL to the code repository for this method (required).
+  repository: https://github.com/saeyslab/CytoNorm
+
+argument_groups:
+  - name: Parameters
+    arguments:
+      - type: integer
+        name: --som_grid_size
+        info:
+          optimize:
+            type: linear
+            lower: 6
+            upper: 16
+        default: 15
+        description: SOM grid size used when training CytoNorm model.
+      - type: integer
+        name: --num_metacluster
+        info:
+          optimize:
+            type: linear
+            lower: 8
+            upper: 20
+        default: 10
+        description: Number of metaclusters generated when training CytoNorm model.
+      - type: integer
+        name: --n_quantiles 
+        info:
+          optimize:
+            type: linear
+            lower: 80
+            upper: 120
+        default: 99
+        description: Number of quantiles to use when training the CytoNorm model. 
+
+# Metadata for your component
+# Resources required to run the component
+resources:
+  # The script of your component (required)
+  - type: r_script
+    path: script.R
+  - path: /src/utils/anndata_to_fcs.R
+
+engines:
+  # Specifications for the Docker image for this component.
+  - type: docker
+    image: openproblems/base_r:1
+    # Add custom dependencies here (optional). For more information, see
+    # https://viash.io/reference/config/engines/docker/#setup .
+    setup:
+      - type: r
+        github: [saeyslab/cytoNorm]
+        bioc: [ flowCore, Biobase ]
+        packages: [ anndata, docstring ]
+
+runners:
+  # This platform allows running the component natively
+  - type: executable
+  # Allows turning the component into a Nextflow module / pipeline.
+  - type: nextflow
+    directives:
+      label: [midtime,midmem,midcpu]

src/methods/cytonorm_all_controls/script.R → src/methods/cytonorm_all_controls_to_mid/script.R

@@ -36,9 +36,22 @@ fset_all <- anndata_to_fcs(adata)
 cat("Setting up some variables for training the model\n")
 
 # get batch label for the training data
-batch_lab_train <- sapply(sampleNames(fset_train), function(samp) {
-    unique(adata[adata$obs$sample == samp]$obs$batch)[1]
-})
+batch_lab_train <- vapply(sampleNames(fset_train), function(samp) {
+    as.character(
+        unique(
+            adata[adata$obs$sample == samp]$obs$batch
+        )[1]
+    )
+}, FUN.VALUE = character(1))
+
+# get batch label for the all data
+batch_labs <- vapply(sampleNames(fset_all), function(samp) {
+    as.character(
+        unique(
+            adata[adata$obs$sample == samp]$obs$batch
+        )[1]
+    )
+}, FUN.VALUE = character(1))
 
 markers_to_correct <- as.vector(adata$var$channel[adata$var$to_correct])
 
@@ -74,11 +87,6 @@ model <- CytoNorm::CytoNorm.train(
     verbose = FALSE
 )
 
-# get batch label for the validation data
-batch_labs <- sapply(sampleNames(fset_all), function(samp) {
-    unique(adata[adata$obs$sample == samp]$obs$batch)[1]
-})
-
 cat("Normalising using Cytonorm model trained using all control samples\n")
 
 norm_fset_all <- CytoNorm::CytoNorm.normalize(

src/methods/cytonorm_no_controls/config.vsh.yaml → src/methods/cytonorm_no_controls_to_goal/config.vsh.yaml

@@ -7,23 +7,25 @@ __merge__: ../../api/comp_method.yaml
 
 # A unique identifier for your component (required).
 # Can contain only lowercase letters or underscores.
-name: cytonorm_no_controls
+name: cytonorm_no_controls_to_goal
 # A relatively short label, used when rendering visualisations (required)
-label: CytoNorm without controls
+label: CytoNorm (no-controls, to-goal)
 # A one sentence summary of how this method works (required). Used when 
 # rendering summary tables.
-summary: CytoNorm without control samples.
+summary: CytoNorm run without control samples, correcting to a goal batch.
 # A multi-line description of how this component works (required). Used
 # when rendering reference documentation.
 description: |
   CytoNorm corrects batch effects by using reference control samples (aliquots of one sample, 
   technical replicates) included with each batch. 
-  It clusters cells, then trains a model on the control samples to learn how marker 
+  It clusters cells using FlowSOM, then trains a model on the control samples to learn how marker 
   expression distributions differ across batches for each population.
-  It then uses splines to align these distributions to a common reference (either the mean
-  of batches or to a single batch).
+  It then uses splines to align these distributions to a common reference (either a midpoint
+  derived from all batches or to a single batch).
+
   In this CytoNorm version, an aggregate of each batch is created and subsequently used as a 
-  proxy for the control samples.
+  proxy for the control samples, aligning the batches to batch 1.
+
   The number of cells used to create an aggregate is set as the number of cells in the smallest 
   sample or 1,000,000, whichever is the smaller, multiplied by how many samples there are in the batch.
   Clustering was performed by FlowSOM.