Skip to content

Optimize: add post-download population subsetting to reduce processing overhead #1

Description

@balis

Context

The 1000 Genomes Phase 3 FTP server only provides whole-cohort VCF files (ALL.chr*.vcf.gz)
containing all 2504 samples. No population-specific files exist. This was verified by checking
every relevant directory on the server:

  • /release/20130502/ — only ALL.chr*
  • /supporting/bcf_files/ — only ALL.chr*
  • /supporting/vcf_with_sample_level_annotation/ — only ALL.chr*
  • /supporting/related_samples_vcf/ — only ALL.chr*

Problem

When an analysis targets specific populations (e.g., only EUR and AFR), the workflow still:

  1. Downloads all 2504 samples (unavoidable — server limitation)
  2. Processes all 2504 sample columns through individuals.py before filtering

Proposed optimization

Add a bcftools view -S <samples.txt> step immediately after data extraction (tabix/download)
to subset the VCF to only the requested populations' samples before further processing.
The panel file (integrated_call_samples_v3.20130502.ALL.panel) provides sample-to-population
mappings.

This would reduce disk usage and processing time in downstream steps (individuals.py,
mutation_overlap.py, frequency.py) proportionally to the population subset ratio.

Impact

  • Network transfer: No change (unavoidable)
  • Disk usage: Reduced proportionally (e.g., 2 of 5 super-populations → ~60% reduction)
  • Processing time: Reduced in individuals/mutation_overlap/frequency steps

Notes

  • The columns.txt balanced sampling (30 samples/population) already limits processing
    for the current test setup, but full-scale runs would benefit significantly
  • The subsetting step could be added to data_preparation.steps in the plan as a
    post-extraction command

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions