Surviving host immune pressures, environmental shifts, and antimicrobial interventions requires a bacterial pathogen to possess relentless genetic plasticity. Most bacterial genes do not operate in isolation; they form elaborate networks whose essentiality is heavily influenced by the background composition of the accessory genome. Understanding the structural architecture of these networks -- how genes are gained, lost, rearranged, and compartmentalized across clinical lineages -- is a prerequisite for deciphering the genotype-to-phenotype map of a major human pathogen.
This repository provides a fully automated, reproducible bioinformatics pipeline that constructs a high-resolution, partitioned pangenome graph of Streptococcus pneumoniae. The work serves as a strict methodological replication of the structural genomic analysis foundational to the findings reported in Rosconi et al., "A bacterial pan-genome makes gene essentiality strain-dependent and evolvable" (Nature Microbiology, 2022). Building on the original study's insight that gene essentiality in S. pneumoniae is profoundly context-dependent, this pipeline extends the analytical scope by incorporating quality control verification (FastANI), maximum-likelihood phylogenomics (IQ-TREE), and comprehensive resistome profiling (AMRFinderPlus), in addition to the core pangenome graph construction.
While large-scale epidemiological studies frequently utilize thousands of draft assemblies, the highly fragmented nature of short-read sequencing obscures the structural reality of horizontal gene transfer and synteny. To achieve unbroken mapping of the accessory genome -- specifically regarding mobile genetic elements such as plasmids, phages, and transposons -- this replication is deliberately restricted to a curated cohort of 36 completely closed genomic assemblies from NCBI RefSeq, mimicking the fidelity of the original PacBio SMRT-sequencing dataset.
The pipeline is organized into six analytical phases, each designed to address a specific dimension of pangenomic characterization.
Automated retrieval of 36 complete S. pneumoniae genomes from the NCBI RefSeq database using the NCBI Datasets CLI (TaxID 1313). A network-resilient download loop with retry logic ensures robustness against transient API failures. All assemblies are standardized to a uniform nomenclature scheme based on their RefSeq accession identifiers.
Prior to pangenome construction, cohort integrity is verified through all-versus-all pairwise Average Nucleotide Identity (ANI) computation using FastANI. This step statistically confirms that no cryptic species or contaminated assemblies inflate the accessory genome, a critical but frequently overlooked verification in pangenomic studies. ANI values are rendered as both an interactive 3D topographic surface and a 2D identity heatmap using Plotly.
Rapid, high-throughput genomic annotation using Prokka establishes a standardized locus nomenclature across all 36 clinical isolates. This homogenization eliminates annotation bias originating from disparate NCBI submission pipelines, ensuring that gene family clustering downstream reflects genuine biological homology rather than nomenclatural artifacts.
Construction of a partitioned synteny graph using PPanGGOLiN (v2.2.6). Unlike simple presence/absence matrices, PPanGGOLiN employs a statistical framework grounded in graph theory to dynamically partition gene families into three genomic compartments based on their topological conservation and occurrence frequency:
- Persistent genome: Gene families conserved across the vast majority of strains, representing the essential metabolic and structural backbone.
- Shell genome: Moderately distributed gene families that define lineage-specific functional modules and ecological adaptations.
- Cloud genome: Rare gene families, frequently associated with mobile genetic elements, phage integration events, and recently acquired genetic cargo.
The panrgp workflow further delineates Regions of Genomic Plasticity (RGPs) and identifies localized insertion hotspots within the persistent backbone, providing a structural map of the zones most susceptible to horizontal gene transfer.
The pangenome matrix is ultimately constrained by shared descent. To contextualize the structural variation within an evolutionary framework, a maximum-likelihood (ML) phylogeny is inferred from a concatenated core-genome protein alignment:
- Extraction: Persistent gene families are extracted from the PPanGGOLiN
.h5binary pangenome using theppanggolin msamodule, ensuring that only vertically inherited core components inform the phylogeny. - Alignment: Multiple sequence alignment (MSA) is performed on the concatenated protein sequences using MAFFT.
- Tree inference: IQ-TREE computes the optimal phylogenetic topology using the LG general-purpose protein substitution model with 1,000 ultrafast bootstrap replicates for branch support assessment.
The resulting cladogram serves as the evolutionary scaffold onto which accessory genome dynamics and functional ontologies are overlaid.
AMRFinderPlus is deployed across the entire cohort to systematically catalogue acquired antimicrobial resistance genes, point mutations conferring resistance, and heavy metal resistance determinants. Resistance profiles are intersected with the structurally variable regions (Shell and Cloud genomes) to identify hyper-mobile genetic elements potentially driving the dissemination of multi-drug resistance cassettes. Results are rendered as an interactive binary heatmap linking strain identity to resistance gene presence.
Beyond standard pangenome metrics, this project includes a comprehensive analytical visualization suite built with Plotly, Scikit-learn, Seaborn, and Toytree. All interactive HTML plots can be opened in any standard web browser.
| Visualization | Description |
|---|---|
| 3D ANI Topography | Surface plot of pairwise Average Nucleotide Identity values across the cohort, revealing the degree of genomic coherence. |
| ANI Identity Heatmap | 2D heatmap of the ANI adjacency matrix, enabling rapid identification of divergent or misclassified strains. |
| 3D Accessory Genome PCA | Principal Component Analysis of the binary gene presence/absence matrix, coupled with K-Means clustering (k=3), compressing the accessory genome into three dimensions to isolate hidden sub-lineages and HGT outbreak clusters. |
| 3D Strain Composition Space | Spatial mapping of each strain based on its Persistent, Shell, and Cloud family counts, enabling visual identification of hyper-plastic outliers with disproportionate accessory content. |
| Stacked Genome Composition | Ordered stacked bar chart demonstrating that variations in genome size are driven almost entirely by accessory genome expansion, while the persistent core remains conserved. |
| U-Shaped Distribution | Classic pangenomic frequency distribution plot generated by PPanGGOLiN, partitioning gene families by their prevalence across the cohort. |
| Tile Plot | PPanGGOLiN-generated tile plot providing a per-genome, per-gene-family visualization of the pangenome partitioning. |
| Global Composition Pie Chart | Proportional distribution of Persistent, Shell, and Cloud gene families across the entire pangenome. |
| Partition Variability Boxplot | Distribution of gene counts per partition across all strains, quantifying the variance in each genomic compartment. |
| Core-Genome ML Phylogeny | Maximum-likelihood phylogenetic tree inferred from the concatenated persistent genome alignment, rendered via Toytree. |
| Accessory Genome Clustermap | Hierarchically clustered presence/absence heatmap of variable (Shell/Cloud) gene families across all strains, revealing co-occurrence patterns and lineage-specific gene blocks. |
| Resistome Heatmap | Binary heatmap of antimicrobial resistance gene presence across the cohort, linking AMR determinants to specific strains. |
Nature2022_Spneumo_Pangenome/
|
|-- 01_raw_genomes/ # 36 complete .fna assemblies from NCBI RefSeq
| |-- GCF_000006885.1.fna
| |-- GCF_000007045.1.fna
| |-- ... # (36 files total)
| +-- GCF_003003495.1.fna
|
|-- 02_prokka_annotations/ # Standardized annotations per strain
| |-- GCF_000006885.1/
| | |-- GCF_000006885.1.gff # General Feature Format annotation
| | |-- GCF_000006885.1.faa # Protein FASTA sequences
| | |-- GCF_000006885.1.ffn # Nucleotide CDS sequences
| | |-- GCF_000006885.1.gbk # GenBank flat file
| | |-- GCF_000006885.1.fna # Input genome (copied)
| | |-- GCF_000006885.1.sqn # Sequin submission file
| | |-- GCF_000006885.1.tbl # Feature table
| | |-- GCF_000006885.1.tsv # Annotation statistics
| | |-- GCF_000006885.1.log # Prokka log
| | |-- GCF_000006885.1.err # Prokka warnings
| | |-- GCF_000006885.1.fsa # Scaffold FASTA
| | +-- GCF_000006885.1.txt # Summary statistics
| |-- GCF_000007045.1/
| +-- ... # (36 strain directories total)
|
|-- 03_ppanggolin_output/ # PPanGGOLiN pangenome graph and analytics
| |-- annotation_map.tsv # Strain-to-GFF mapping file for PPanGGOLiN
| +-- pangenome_graph/
| |-- pangenome.h5 # Binary pangenome database (HDF5)
| |-- pangenomeGraph.gexf # Full graph (GEXF format, for Gephi)
| |-- pangenomeGraph.json # Full graph (JSON format)
| |-- pangenomeGraph_light.gexf # Lightweight graph
| |-- gene_families.tsv # Gene family assignments
| |-- gene_presence_absence.Rtab # Binary presence/absence matrix
| |-- matrix.csv # Annotated partition matrix
| |-- genomes_statistics.tsv # Per-genome partition statistics
| |-- mean_persistent_duplication.tsv # Core gene duplication metrics
| |-- tile_plot.html # PPanGGOLiN tile plot
| |-- Ushaped_plot.html # PPanGGOLiN U-shaped distribution
| |-- Interactive_3D_PCA.html # 3D PCA scatter (K-Means)
| |-- Interactive_3D_Composition.html # 3D strain composition scatter
| |-- Interactive_Stacked_Composition.html # Stacked bar chart
| |-- partitions/ # Per-partition gene family lists
| | |-- persistent.txt
| | |-- shell.txt
| | |-- cloud.txt
| | +-- ...
| |-- gff/ # PPanGGOLiN-annotated GFF files per strain
| |-- proksee/ # Proksee JSON visualizations per strain
| +-- table/ # Per-strain partition TSV tables
|
|-- 04_dataset/ # Packaged dataset for cloud execution
| +-- Spneumoniae_Nature2022_Dataset.zip # Compressed raw genomes dataset archive
|
|-- bio_env/ # Conda/Mamba bioinformatics environment (gitignored)
| +-- README.md # Placeholder with recreation instructions
|
|-- Nature2022_Spneumo_Pangenome_Analysis.ipynb # Pre-configured Jupyter notebook (Cloud-ready)
|-- installation.sh # Conda/Mamba environment construction script
|-- pipeline.sh # Master bash pipeline (local execution)
|-- visualization.py # PCA, K-Means, and Plotly HTML generation
|-- .gitignore # Excludes bio_env/ and temporary files
+-- README.md # This document
This project supports two execution modes: local pipeline execution and cloud-based notebook execution.
All bioinformatics dependencies are managed through an isolated Conda/Mamba environment. The environment construction is fully automated via installation.sh and requires either Conda or Mamba to be available on the system PATH.
chmod +x installation.sh
./installation.sh
conda activate ./bio_envThe following tools are installed into the environment:
| Tool | Version | Purpose |
|---|---|---|
| NCBI Datasets CLI | latest | Genome retrieval from RefSeq |
| Prokka | latest | Rapid prokaryotic genome annotation |
| PPanGGOLiN | 2.2.6 | Partitioned pangenome graph construction |
| FastANI | latest | Average Nucleotide Identity computation |
| MAFFT | latest | Multiple sequence alignment |
| IQ-TREE | latest | Maximum-likelihood phylogenetic inference |
| AMRFinderPlus | latest | Antimicrobial resistance gene detection |
| Plotly | latest | Interactive visualization rendering |
| Scikit-learn | latest | PCA and K-Means clustering |
| Pandas | latest | Data manipulation and analysis |
The master pipeline script handles genome download, Prokka annotation, PPanGGOLiN graph construction, and dataset staging in a single invocation.
chmod +x pipeline.sh
./pipeline.shFollowing successful pangenome graph construction, the visualization script generates the interactive HTML analytics from the output matrices.
python visualization.pyThe resulting .html files in 03_ppanggolin_output/pangenome_graph/ are fully interactive and can be opened in any standard web browser.
For cloud-based reproducibility, the pre-configured Jupyter notebook (Nature2022_Spneumo_Pangenome_Analysis.ipynb) is designed for execution on Hugging Face / interactive environments. It installs its own isolated Micromamba environment, retrieves data directly from the companion Hugging Face dataset, and executes the full analytical pipeline (Modules 1 through 5).
This includes FastANI quality control, Prokka annotation, PPanGGOLiN graph partitioning, IQ-TREE phylogenomics, AMRFinderPlus resistome profiling, and all interactive visualizations. The live notebook can be viewed and executed directly on .
The 36 complete Streptococcus pneumoniae genome assemblies used in this study were retrieved from the NCBI RefSeq database (Taxonomy ID: 1313, Assembly Level: Complete). The curated dataset is also available as a Hugging Face dataset for direct use with the companion notebook:
- Hugging Face dataset: S. pneumoniae Structural Pangenomics Cohort
The following NCBI RefSeq accessions comprise the cohort:
Click to expand accession list (36 genomes)
| # | Accession |
|---|---|
| 1 | GCF_000006885.1 |
| 2 | GCF_000007045.1 |
| 3 | GCF_000014365.2 |
| 4 | GCF_000018965.1 |
| 5 | GCF_000018985.1 |
| 6 | GCF_000019005.1 |
| 7 | GCF_000019025.1 |
| 8 | GCF_000019265.1 |
| 9 | GCF_000019985.1 |
| 10 | GCF_000026665.1 |
| 11 | GCF_000147095.1 |
| 12 | GCF_000196595.1 |
| 13 | GCF_000210935.1 |
| 14 | GCF_000210955.1 |
| 15 | GCF_000210975.1 |
| 16 | GCF_000210995.1 |
| 17 | GCF_000211015.1 |
| 18 | GCF_000211035.1 |
| 19 | GCF_000211055.1 |
| 20 | GCF_000211075.1 |
| 21 | GCF_000251085.2 |
| 22 | GCF_000273445.1 |
| 23 | GCF_000299015.1 |
| 24 | GCF_001095405.2 |
| 25 | GCF_001255215.1 |
| 26 | GCF_001457635.1 |
| 27 | GCF_001896045.1 |
| 28 | GCF_001896065.1 |
| 29 | GCF_001896085.1 |
| 30 | GCF_001902455.1 |
| 31 | GCF_002357995.1 |
| 32 | GCF_002813535.1 |
| 33 | GCF_002813955.1 |
| 34 | GCF_002843545.1 |
| 35 | GCF_002947575.1 |
| 36 | GCF_003003495.1 |
This project is provided for academic and research purposes. The underlying data are publicly available through NCBI RefSeq.
Rosconi, F., Rudmann, E., Li, J., Surujon, D., Anthony, J., Frank, M., ... & van Opijnen, T. (2022). A bacterial pan-genome makes gene essentiality strain-dependent and evolvable. Nature Microbiology, 7(10), 1580--1592. https://doi.org/10.1038/s41564-022-01208-7