Context
The 1000 Genomes Phase 3 FTP server only provides whole-cohort VCF files (ALL.chr*.vcf.gz)
containing all 2504 samples. No population-specific files exist. This was verified by checking
every relevant directory on the server:
/release/20130502/ — only ALL.chr*
/supporting/bcf_files/ — only ALL.chr*
/supporting/vcf_with_sample_level_annotation/ — only ALL.chr*
/supporting/related_samples_vcf/ — only ALL.chr*
Problem
When an analysis targets specific populations (e.g., only EUR and AFR), the workflow still:
- Downloads all 2504 samples (unavoidable — server limitation)
- Processes all 2504 sample columns through
individuals.py before filtering
Proposed optimization
Add a bcftools view -S <samples.txt> step immediately after data extraction (tabix/download)
to subset the VCF to only the requested populations' samples before further processing.
The panel file (integrated_call_samples_v3.20130502.ALL.panel) provides sample-to-population
mappings.
This would reduce disk usage and processing time in downstream steps (individuals.py,
mutation_overlap.py, frequency.py) proportionally to the population subset ratio.
Impact
- Network transfer: No change (unavoidable)
- Disk usage: Reduced proportionally (e.g., 2 of 5 super-populations → ~60% reduction)
- Processing time: Reduced in individuals/mutation_overlap/frequency steps
Notes
- The
columns.txt balanced sampling (30 samples/population) already limits processing
for the current test setup, but full-scale runs would benefit significantly
- The subsetting step could be added to
data_preparation.steps in the plan as a
post-extraction command
Context
The 1000 Genomes Phase 3 FTP server only provides whole-cohort VCF files (
ALL.chr*.vcf.gz)containing all 2504 samples. No population-specific files exist. This was verified by checking
every relevant directory on the server:
/release/20130502/— onlyALL.chr*/supporting/bcf_files/— onlyALL.chr*/supporting/vcf_with_sample_level_annotation/— onlyALL.chr*/supporting/related_samples_vcf/— onlyALL.chr*Problem
When an analysis targets specific populations (e.g., only EUR and AFR), the workflow still:
individuals.pybefore filteringProposed optimization
Add a
bcftools view -S <samples.txt>step immediately after data extraction (tabix/download)to subset the VCF to only the requested populations' samples before further processing.
The panel file (
integrated_call_samples_v3.20130502.ALL.panel) provides sample-to-populationmappings.
This would reduce disk usage and processing time in downstream steps (
individuals.py,mutation_overlap.py,frequency.py) proportionally to the population subset ratio.Impact
Notes
columns.txtbalanced sampling (30 samples/population) already limits processingfor the current test setup, but full-scale runs would benefit significantly
data_preparation.stepsin the plan as apost-extraction command