diff --git a/.gitignore b/.gitignore index 01ccbf511..9ef43dddb 100644 --- a/.gitignore +++ b/.gitignore @@ -25,3 +25,6 @@ lineage/data/taxol output/ docs/_build/ + +# KP-Tracer analysis intermediates (large; regenerate with analysis/kptracer scripts) +analysis/kptracer/results/*.pkl diff --git a/analysis/kptracer/README.md b/analysis/kptracer/README.md new file mode 100644 index 000000000..1bb1f298f --- /dev/null +++ b/analysis/kptracer/README.md @@ -0,0 +1,34 @@ +# Latent-state tHMM on KP-Tracer phylogenies (issue #1014) + +The findings are in [`REPORT.md`](REPORT.md). This directory holds the scripts that produce them. + +## Data + +Download the processed KP-Tracer release (Yang et al. 2022, Zenodo record 5847462, CC-BY, 1.3 GB): + +```sh +mkdir -p ~/data/kptracer && cd ~/data/kptracer +curl -L -o KPTracer-Data.tar.gz https://zenodo.org/api/records/5847462/files/KPTracer-Data.tar.gz/content +tar xzf KPTracer-Data.tar.gz +``` + +The scripts look for `~/data/kptracer/KPTracer-Data` or `$KPTRACER_DATA`. + +## Pipeline + +Run from the repository root. `anndata` is only needed to read the h5ad, and `matplotlib` only for figures. + +```sh +uv run python analysis/kptracer/simulation_study.py # results/simulation.csv +uv run python analysis/kptracer/scalability.py # results/scalability.csv (run from this dir) +uv run --with anndata python analysis/kptracer/prepare_data.py # results/kptracer_inputs.pkl +cd analysis/kptracer +uv run python kptracer_analysis.py select # K and branch-length selection (~45 min on 46 cores) +uv run python kptracer_analysis.py fit 12 # anchored (fixed emissions), unsupervised K=12, sensitivity fits +uv run python kptracer_analysis.py bootstrap anchored +uv run python kptracer_analysis.py compare +uv run --with matplotlib python make_figures.py +uv run python bdmm_comparison.py # tHMM side of the BDMM-Prime comparison (bdmm/ holds the BEAST XML) +``` + +`results/*.pkl` are large intermediates and are git-ignored. The CSV summaries and `figures/` are committed. diff --git a/analysis/kptracer/REPORT.md b/analysis/kptracer/REPORT.md new file mode 100644 index 000000000..748b298c9 --- /dev/null +++ b/analysis/kptracer/REPORT.md @@ -0,0 +1,226 @@ +# Latent-state tHMM on CRISPR lineage-recorder phylogenies: KP-Tracer report (#1014) + +## Summary + +The tHMM now fits reconstructed phylogenies directly. Each edge has its own transition matrix exp(Q·t_e). Hidden states have Gaussian emissions on a cell embedding and are observed only at the leaves. Q is estimated by an exact EM step. In simulation this model recovers rates, emissions and ancestral states even at 10% leaf sampling. Its expected numbers of state switches are unbiased. Cluster-then-parsimony instead over-counts switches by 1.4–2.9× when states overlap, and under-counts them even when given the true labels. + +On 21 KP-Tracer sgNT primary tumors (28,939 cells), the tHMM reproduces the published tumor-level plasticity ranking. Its Spearman ρ is 0.94 against Fitch parsimony and 0.94 against the published mean scPlasticity. It also reproduces the paper's qualitative picture: AT2-like is the most likely root state and is stable, while High-plasticity, AT1-like and the gastric/endoderm states are the most plastic. + +The model adds three things parsimony does not provide: + +1. **Direction of switches.** Some flows are strongly directed and robust to resampling tumors: Early gastric → AT1-like, Gastric-like → AT1-like, Late gastric → Endoderm-like, and Early EMT-1 → Pre-EMT. Others that Fitch reports as one-way, AT2 ↔ AT1 and Gastric-like ↔ High-plasticity, are bidirectional under the model. Yang et al. note they "were limited in our ability to describe the directionality of transitions". +2. **A lineage information gain.** Knowing the tree improves the held-out prediction of a masked cell's expression by about 1.5 nats per cell. +3. **Posterior uncertainty** for every ancestor and edge. + +**Main limitations:** + +- Only two to four tumors carry the Pre-EMT, Late gastric, Early EMT-2 and Lung-progenitor states. Both Mesenchymal states occur only in 3724_NT_T1, a nearly star-shaped tree, so conclusions about them are weak. +- The unsupervised model allocates states by cell density, not biology. It splits the abundant Mesenchymal cells and merges several small epithelial states. +- Mutation counts on edges carry essentially no information beyond the topology. + +## What was built + +| Issue task | Implementation | +|---|---| +| Tree ingestion | `lineage/tree_io.py`. Reads Newick, networkx or `CassiopeiaTree` into CSR with the root at 0 and nodes in BFS order, and names unnamed nodes. Also prunes to cells with expression, collapses unifurcations (summing branch lengths), aligns leaves to `obs_names`, and computes Cassiopeia-style LCA ancestral alleles and per-edge mutation counts. | +| Multivariate Gaussian emission | `lineage/states/StateDistributionGaussian.py`. Diagonal-plus-ridge covariance. All-NaN rows (internal nodes, masked leaves) have log-likelihood 0; partially missing rows are marginalized. | +| Log-space emissions | `get_log_Emission_Likelihoods` and `get_scaled_Emission_Likelihoods` in `lineage/LineageTree.py`. Subtracting the per-node maximum is exact: it rescales only that node's normalizing factor. The upward pass also needed log-space products (`get_beta_and_logNF`), because KP-Tracer polytomies with thousands of children overflow `np.prod`. | +| Per-edge exp(Qt) and Q M-step | `lineage/HMM/E_step.py` accepts either a shared (K×K) or a per-edge (N×K×K) transition array; the existing code path is unchanged. `lineage/ctmc.py` computes exp(Qt) by eigendecomposition. It also implements the endpoint-conditioned EM of Hobolth & Jensen (2011). The expected dwell times and jump counts need one eigendecomposition per iteration plus O(K²) work per edge. | +| Edge-wise joint posterior and switch counts | `get_edge_posteriors` in `lineage/HMM/M_step.py` gives P(z_p = k, z_c = l \| Y) for every edge. `PhyloHMM.switch_summary` reports expected k→l edge changes, CTMC jumps (including reversions within an edge), per-edge change probabilities, and plasticity and heritability per tree. | +| Model | `lineage/phyloHMM.py` (`PhyloHMM`). EM with k-means initialization and several restarts. Offers `ctmc` or `discrete` (one T per edge, for division-resolved trees) modes, fixed or free emissions, BIC, and held-out leaf likelihood with an optional tree-free baseline. | +| Baselines | `lineage/phylo_stats.py`: unit-cost Sankoff/Fitch–Hartigan small parsimony on multifurcating trees; PATH-style Moran's I with inverse node-distance weights and permutation z-scores. | +| Simulation | `lineage/phylo_sim.py`: Yule trees, CTMC states along branches, Gaussian leaves, leaf subsampling with pruning. | +| Tests | `lineage/tests/test_phyloHMM.py`, 19 tests. They check the E-step against brute-force enumeration of all state assignments on a multifurcating tree (log-likelihood, node marginals, edge joints) in both modes. They check the CTMC statistics against numerical quadrature, a 5,000-child polytomy, underflow at D = 300, the loader, parsimony, EM monotonicity and Q recovery. The existing 67 tests pass unchanged. | + +Analysis scripts, CSV results and figures are in this directory; see [README.md](README.md). + +## Open questions from the issue + +- **Tree format.** `KPTracer-Data/trees/*_tree.nwk` are plain Newick with leaf names only: no branch lengths, and many unifurcations. Each tree has a per-tumor `*_character_matrix.txt` (cells × cut sites, `-` = missing) and a `*_priors.pkl`. Branch lengths therefore come from mutation counts on edges, after LCA reconstruction as in the Cassiopeia release. Many edges carry no new mutation (14–96% per tumor, mostly leaf edges hanging off polytomies), so edge length is t = n_mut + δ. +- **Embedding.** The h5ad provides the 10-dimensional batch-corrected `X_scVI` that the published Leiden clusters were built on, so scVI is the primary embedding. PCA (top 2,000 variable genes, 10 PCs, not batch-corrected) is the sensitivity analysis. The two unsupervised K = 12 fits agree only moderately with each other (ARI 0.57), and about equally with the Leiden labels (NMI 0.71 for scVI, 0.70 for PCA). +- **Published scores.** The Zenodo release includes `plasticity_scores.tsv` with per-cell scPlasticity, used directly below. My own Fitch implementation reproduces the published tumor ranking (ρ = 0.96 against the mean published scPlasticity). + +## Simulation study + +`simulation_study.py`, figures `figures/simulation_accuracy.png` and `figures/simulation_Q_error.png`. + +- **Setup:** 20 Yule trees of 500 leaves each; K = 4 states in a progression-like chain (0 → 1 → 2 → 3 with slow reversions); 10-dimensional Gaussian emissions with unit noise. Means are either well separated (scale 1.0) or overlapping (scale 0.5), and leaf sampling runs from 100% down to 10%. Each setting has 5 replicates; each fit keeps the best of 3 starts. +- **Compared methods:** the tHMM (exp(Qt) per edge, or one T per edge ignoring lengths), k-means labels followed by Fitch parsimony, and true leaf labels followed by Fitch parsimony (an oracle). + +![simulation](figures/simulation_accuracy.png) + +| states | leaves sampled | method | leaf acc. | ancestral acc. | est./true switches | Q rel. error | +|---|---|---|---|---|---|---| +| separated | 100% | tHMM exp(Qt) | 0.98 | **0.98** | **1.00** | 0.09 | +| | | k-means + parsimony | 0.92 | 0.96 | 1.43 | – | +| | | true labels + parsimony | 1.00 | 0.97 | 0.93 | – | +| separated | 10% | tHMM exp(Qt) | 0.96 | **0.93** | **0.96** | 0.18 | +| | | tHMM one T | 0.96 | 0.86 | 0.98 | – | +| | | k-means + parsimony | 0.93 | 0.86 | 0.95 | – | +| | | true labels + parsimony | 1.00 | 0.88 | 0.84 | – | +| overlapping | 100% | tHMM exp(Qt) | **0.90** | 0.94 | **1.00** | 0.09 | +| | | k-means + parsimony | 0.66 | 0.83 | 2.87 | – | +| overlapping | 10% | tHMM exp(Qt) | **0.79** | **0.85** | **0.97** | 0.73 | +| | | tHMM one T | 0.75 | 0.72 | 0.96 | – | +| | | k-means + parsimony | 0.65 | 0.69 | 1.40 | – | +| | | true labels + parsimony | 1.00 | 0.88 | 0.84 | – | + +Observations: + +- **Switch counts are unbiased only for the tHMM.** Parsimony is a minimum, so it under-counts even with the true labels (0.84–0.93×). With estimated labels, every misclassified leaf adds a spurious switch, so it over-counts (up to 2.9×). The error gets *worse* as more leaves are sampled. +- **Ancestral states improve on even oracle parsimony when sampling is sparse.** With separated states and 10% sampling, the tHMM reaches 0.93 against 0.88 for parsimony on true labels. It pools evidence across the subtree and uses branch lengths. +- **Branch lengths matter under sparse sampling.** At 10% sampling, ignoring them (one T per edge) costs 7–13 points of ancestral accuracy. Pruned, subsampled trees have very uneven edge lengths. +- **Q is well identified with dense sampling or separated states** (relative error ≤ 0.18). With overlapping states and 10% sampling it degrades (0.73) because the small rates are poorly determined; the large rates stay within about 30%. + +## Scalability + +`scalability.py`, `figures/scalability.png`. Runtime on one core is linear in tree size. One E-step takes 4 ms on 199 nodes, 0.38 s on 20k nodes and 1.2 s on 60k nodes. A full EM fit takes 0.27 s at 100 leaves and 34 s at 30,000 leaves. The whole KP-Tracer cohort (21 trees, 45k nodes) fits in about 1 minute per start. + +![scalability](figures/scalability.png) + +**Comparison with BDMM-Prime on a 100-cell tree** (`bdmm_comparison.py`, `results/bdmm_thmm.json`). Both methods get the same simulated 100-leaf time tree, the true tip types (4 types), and the same branch lengths. + +- **tHMM:** fits in 0.4 s, with emissions fixed at near-indicators of the tip type. A 200-replicate parametric bootstrap takes 165 s. Its 95% intervals cover all 12 true rates, but they are very wide on a tree this small. The four large rates are recovered in the right order: 0→1 at 0.69 (true 0.45), 1→0 at 0.27 (0.15), 1→2 at 0.26 (0.30), 2→3 at 0.14 (0.24). With only 34 true switches on the tree, the small rates go to the zero boundary. +- **BDMM-Prime** (BEAST 2.7.7, BDMM-Prime 2.7.2; XML and summary in `bdmm/`): + - **Setup:** fixed tree; the standard BDMM likelihood integrated over type histories; shared birth rate; no death; ρ = 1; Exponential(mean 0.3) priors on the 12 type-change rates; estimated root frequencies. + - **Runtime:** about 2.2–2.6 ms per MCMC state. Mixing was excellent (single-chain ESS > 2,400 over 2.5M states), so ESS > 200 needs about 200k states, roughly 7–10 minutes for one chain. The full 4-chain run took 107 minutes. + - **Accuracy:** its 95% HPDs cover all 12 true rates, but they are wide. The rates out of the rare type 3 sit near the prior median (0.21–0.27 against true 0.015–0.03), and the root type is not identified. + +| rate | true | tHMM MLE [bootstrap 95%] | BDMM-Prime median [95% HPD] | +|---|---|---|---| +| 0→1 | 0.45 | 0.69 [0.01, 3.1] | 0.51 [0.10, 0.95] | +| 1→0 | 0.15 | 0.27 [0.00, 2.5] | 0.29 [0.04, 0.59] | +| 1→2 | 0.30 | 0.26 [0.00, 3.4] | 0.20 [0.03, 0.41] | +| 2→3 | 0.24 | 0.14 [0.00, 2.7] | 0.13 [0.00, 0.38] | +| rates out of type 3 | 0.015–0.03 | 0.00–0.01 | 0.21–0.27 (prior-driven) | + + On the same information the two methods are about equally accurate for the well-determined rates. The tHMM is about 1,000× faster than even a minimal BDMM-Prime run. It also scales linearly to the 14k-cell KP-Tracer trees, where BDMM-type inference is not practical. The trade-offs: + - The tHMM conditions on the tree and does not model type-dependent birth or death, which BDMM does. + - Its intervals come from a parametric bootstrap, not a posterior. On 100 cells they are much wider than the HPDs, because each refit of a 12-rate model to about 34 switches occasionally explodes one rate, and nothing like BDMM-Prime's prior holds rates down. + - Its maximum-likelihood rates for rarely visited states sit on the zero boundary, where BDMM-Prime's are prior-driven. + +## KP-Tracer analysis + +### Data + +- **Tumors:** filtered as in Yang et al. (Figure 4 notebook of KPTracer-release): sgNT primary tumors with a tree, ≥100 cells, >5% unique alleles and >20% unsaturated targets. This leaves 21 tumors. +- **Cells:** leaves are restricted to cells present in `adata_processed.nt.h5ad`. Leiden clusters holding ≤2.5% of a tumor are dropped, and unifurcations are collapsed. This leaves 28,939 cells in trees of 188 to 14,817 nodes. +- **3724_NT_T1** contributes half of the cells (14,480) but is almost a star: 96% of its edges carry no new mutation. + +### Model selection + +`kptracer_analysis.py select`, `figures/model_selection.png`, `results/model_selection.csv`. For each K from 2 to 14 there are 3 replicates. Each replicate masks 20% of leaves, fits the model, and scores the masked cells by their predictive density. The **lineage information gain** compares that score against a tree-free prediction that uses the same emissions but population-wide state frequencies. + +![selection](figures/model_selection.png) + +- Held-out likelihood and BIC improve monotonically up to K = 14. As expected for a mixture on a continuous embedding, neither picks a small K. +- The lineage gain is large at every K: 3.8k nats at K = 2 and 8.8k at K = 12, over about 5.8k masked cells. Returns diminish past K ≈ 12, so the unsupervised model uses K = 12. +- **Branch lengths.** Held-out likelihood increases steadily from t = n_mut + 0.1 through n_mut + 2 to unit lengths (t = 1 on every edge), which score best. The differences are small (≤ 220 nats, against a lineage gain of about 8,000). The fixed-emission fits agree: the log-likelihood is −303.0k for δ = 0.1, −302.5k for δ = 0.5, −302.1k for δ = 2 and −302.0k for unit lengths. So the number of recorder mutations on an edge is not a useful proxy for how many divisions or how much time the edge spans. All tumor-level conclusions below hold across these choices (see the sensitivity table). + +### Cluster-anchored model: published states, learned dynamics + +To compare with Yang et al. on their own state definitions, there is one hidden state per published Leiden cluster (13 remain after filtering). **Emissions are fixed** at each cluster's Gaussian in scVI space, and Q and π are learned. Emissions must be fixed: letting them move (`anchored_freeE`) makes EM repurpose states. The Pre-EMT Gaussian absorbs half of the Mesenchymal-2 cells in 3724_NT_T1, and the Endoderm-like state ends up covering none of its own cells. The share of cells whose most likely state matches their own cluster then falls from 0.94 (fixed emissions) to 0.70. + +**Tumor-level plasticity.** Plasticity is the expected number of edge state changes divided by the number of edges; `results/tumor_benchmarks_anchored.csv`. + +![benchmarks](figures/tumor_benchmarks_anchored.png) + +| comparison (21 tumors, Spearman ρ) | main (δ = 0.5) | without 3724_NT_T1 | δ = 0.1 | δ = 2 | unit lengths | free emissions | +|---|---|---|---|---|---|---| +| tHMM vs. Fitch plasticity | **0.94** | 0.93 | 0.92 | 0.95 | 0.95 | 0.52 | +| tHMM vs. published mean scPlasticity | **0.94** | 0.93 | 0.88 | 0.93 | 0.91 | 0.46 | +| Fitch (ours) vs. published | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | 0.96 | +| tHMM vs. PATH-style auto-correlation z | −0.15 | −0.01 | −0.15 | −0.10 | −0.15 | 0.10 | +| Fitch vs. PATH-style auto-correlation z | −0.01 | 0.13 | −0.01 | −0.01 | −0.01 | −0.01 | +| per-cell tHMM vs. published scPlasticity (median within tumor) | 0.45 | 0.44 | 0.47 | 0.34 | 0.45 | 0.38 | +| pooled k→l matrix, tHMM vs. Fitch | 0.79 | 0.74 | 0.78 | 0.78 | 0.78 | 0.65 | +| total edge changes, tHMM / Fitch | 4,125 / 3,888 | 4,029 / 3,753 | 4,372 / 3,888 | 3,717 / 3,888 | 3,518 / 3,888 | 8,228 / 3,888 | +| total CTMC jumps | 7,321 | 7,305 | 25,550 | 4,732 | 4,152 | 11,869 | + +- tHMM plasticity tracks both parsimony and the published scores closely. The expected number of edge changes is about 6% above the Fitch minimum, as the simulations predict. The CTMC jump count is higher still because it includes reversions within an edge. It depends strongly on δ, since short edges force the model to explain changes with high rates. Jump counts are therefore not interpretable in absolute terms without a time-calibrated tree. +- PATH-style phylogenetic auto-correlation measures something else. Its z-score mostly tracks tumor size (ρ = 0.54 with cell count) and whether states segregate into whole clades. For example, 3724_NT_T1 has near-zero per-edge plasticity and z ≈ 3,300. Neither parsimony nor the tHMM per-edge plasticity is related to it across tumors. Per-edge switching (Fitch, tHMM) and phylogenetic correlation (PATH) should not be treated as interchangeable measures of "plasticity". + +**Per-state stability.** `results/state_table_anchored.csv`. The table shows P(child = k | parent = k) pooled over edges, with 95% intervals from 48 tumor-bootstrap refits. + +| state | cells (tumors) | tHMM stay prob. [95% CI] | Fitch stay | published scPlasticity | root posterior [95% CI] | +|---|---|---|---|---|---| +| Mesenchymal-1 | 5,527 (1) | 0.997 [0.10, 1.00] | 0.995 | 0.002 | 0.05 | +| Mesenchymal-2 | 8,953 (1) | 0.990 [0.11, 0.99] | 0.988 | 0.009 | 0.00 | +| Lung progenitor-like | 880 (4) | 0.975 [0.55, 1.00] | 0.928 | 0.158 | 0.00 | +| **AT2-like** | 3,884 (19) | **0.915 [0.88, 0.94]** | 0.889 | 0.115 | **0.47 [0.04, 0.70]** | +| Late Gastric | 675 (2) | 0.816 [0.28, 0.83] | 0.755 | 0.227 | 0.00 | +| Gastric-like | 2,747 (13) | 0.809 [0.50, 0.87] | 0.784 | 0.276 | 0.03 | +| Early EMT-1 | 1,010 (5) | 0.804 [0.57, 0.91] | 0.846 | 0.189 | 0.08 | +| Endoderm-like | 476 (7) | 0.749 [0.24, 0.78] | 0.719 | 0.293 | 0.00 | +| Early gastric | 1,717 (6) | 0.735 [0.39, 0.75] | 0.735 | 0.289 | 0.21 | +| **High plasticity** | 1,465 (18) | **0.580 [0.43, 0.65]** | 0.593 | 0.250 | 0.02 | +| **AT1-like** | 1,408 (14) | **0.474 [0.28, 0.57]** | 0.795 | 0.206 | 0.02 | +| Pre-EMT | 165 (2) | 0.427 [0.00, 0.50] | 0.750 | 0.156 | 0.07 | +| Early EMT-2 | 32 (2) | 0.267 [0.00, 0.54] | 0.714 | 0.197 | 0.06 | + +- The ranking agrees with Fitch (ρ = 0.81) and runs opposite to published per-cell plasticity (ρ = −0.48), as it should. It also matches the paper's statements: AT2-like and Mesenchymal "represented the most stable states", and High-plasticity, Gastric-like and Endoderm-like "exhibited high EffectivePlasticity". +- **AT2-like** is the most likely state of the tumor MRCA (0.47, the largest mass). It is also one of the most stable states that is supported by many tumors (19). This fits the paper's "initial, stable alveolar-type2-like state". +- **AT1-like** is much less stable under the tHMM (0.47) than under Fitch (0.80). Most of its switches are on the AT2 ↔ AT1 and Early gastric → AT1 axes (flow table below). Fitch's minimum-change labeling can absorb such isolated leaf-level switches into a single change at a polytomy, which likely explains the gap. +- **Rare states:** intervals are wide for states present in one or two tumors. The Mesenchymal stability estimate rests entirely on 3724_NT_T1. + +**Direction of transitions.** `results/flows_anchored.csv`, `figures/transitions_anchored.png`. The table lists pooled expected CTMC jumps in each direction and the fraction of tumor-bootstrap refits that agree on the net direction. + +| flow | forward | reverse | bootstrap agreement | +|---|---|---|---| +| Early gastric → AT1-like | 686 | 38 | **0.98** | +| Late Gastric → Endoderm-like | 293 | 165 | **0.98** | +| Mesenchymal-2 → Mesenchymal-1 | 86 | 15 | 0.94 (1 tumor) | +| Early EMT-1 → Pre-EMT | 449 | 323 | 0.92 | +| Gastric-like → AT1-like | 124 | 0 | 0.92 | +| Endoderm-like → High plasticity | 101 | 32 | 0.79 | +| AT2-like → High plasticity | 146 | 54 | 0.77 | +| Gastric-like → Early EMT-1 | 57 | 1 | 0.71 | +| AT1-like ↔ AT2-like | 900 | 835 | 0.65 (bidirectional) | +| Gastric-like ↔ High plasticity | 1,139 | 975 | 0.60 (bidirectional) | + +![transitions](figures/transitions_anchored.png) + +- **Fitch direction can be an artifact.** Parsimony on the same labels reports AT2 → AT1 422 times and AT1 → AT2 only 9 times. That direction most likely comes from its tie-breaking, which keeps the parent's label and roots the tree in the majority state. The tHMM finds that axis bidirectional. For Early gastric → AT1-like the two methods agree (568 vs. 19 for Fitch). Both are consistent with the paper's observation that "AT1-like and Early Gastric states clustered together". +- **The paper's route toward EMT.** It proposes that tumors leave AT2 through gastric/endoderm-like or lung-mixed states and then enter EMT. The model's net flows point that way: AT2-like → High plasticity, Endoderm-like → High plasticity, Gastric-like → Early EMT-1, and Early EMT-1 → Pre-EMT. Only Early EMT-1 → Pre-EMT clears 0.9 bootstrap agreement, and the EMT states occur in only 2–5 tumors. These are hypotheses to test on more tumors, not established paths. + +### Unsupervised model (K = 12) + +`fit_unsup`, `figures/unsup_vs_clusters.png`. Agreement with the Leiden labels is NMI 0.71 (ARI 0.51). + +- **Recovered as distinct states:** AT2-like (with 0.63 of the root mass), Early gastric, Gastric-like, Lung-progenitor and Early EMT. +- **Split:** six of the 12 states divide the Mesenchymal cells of 3724_NT_T1, and three of those sub-states switch rapidly (exit rates 1.2–1.8). +- **Merged:** AT1-like, High-plasticity, Endoderm-like and Late gastric collapse into one state. + +Without 3724_NT_T1, the same fit separates High-plasticity, Late gastric/Endoderm, Lung-progenitor and AT1-like into their own states. Maximum-likelihood state discovery puts states where cells are dense, not where biology is resolved. For discovery on data like this, the most useful next steps are per-tumor weighting, hierarchical K, or anchoring on known programs. + +![unsup](figures/unsup_vs_clusters.png) + +## Conclusions + +1. **The generalization is correct and fast.** Per-edge exp(Qt) transitions, leaf-only Gaussian emissions and exact CTMC EM make the tHMM a practical model for recorder phylogenies. It matches brute-force enumeration exactly and scales linearly, taking about 1 minute per start for 29k cells. +2. **It beats cluster-then-parsimony.** In simulation it gives unbiased switch counts and better ancestral states, especially under the heavy leaf subsampling typical of recorder data. +3. **It reproduces the published results.** On KP-Tracer, tumor-level plasticity matches the published scores (ρ = 0.94), AT2-like is identified as the stable root state, and the states the paper calls highly plastic come out as the least stable. +4. **Directionality is the clearest new result.** A non-reversible Q separates robustly directed flows (Early gastric → AT1-like, Late gastric → Endoderm-like, Early EMT-1 → Pre-EMT) from bidirectional ones (AT1 ↔ AT2, Gastric-like ↔ High-plasticity). Parsimony misassigns direction on the latter. The source paper said this was beyond its methods. +5. **Caveats on KP-Tracer:** + - Mutation-count branch lengths are uninformative: unit lengths fit as well or better. + - Several states are supported by only one to four tumors. + - Unsupervised state discovery is density-driven. + - Absolute rates and CTMC jump counts depend on the branch-length model and need a time-calibrated tree before they can be read as per-division or per-day rates. + +## Suggested next steps + +- Replace n_mut + δ with branch lengths estimated jointly with the tree, e.g. Cassiopeia's `IIDExponentialMLE` on the character matrices. Alternatively, learn a per-tumor length scale inside the EM. +- Weight tumors equally in the Q M-step, or use a hierarchical Q, so that one 14k-cell tumor does not dominate the pooled rates. +- Extend to the metastatic families (`*_Fam` trees) and to the KPL/KPA genotypes, which the paper reports have new trajectories. Test whether Q differs by genotype with a likelihood-ratio or bootstrap test. +- Run the spatial extension (deferred here) on Zenodo 19771805, and validate on the division-resolved baseMEMOIR trees using the existing `discrete` mode. + +## Not done + +- **Spatial stretch goals:** skipped, as requested. +- **baseMEMOIR validation (stretch):** not attempted. It needs 290 GB of raw images and segmentation to obtain states. +- **PATH:** the PATH R package was not run. Its Moran's I statistic was reimplemented (`phylo_stats.py`) with inverse node-distance weights and a permutation null. +- **Published per-cell plasticity script:** reading more of KPTracer-release's `compute_plasticity_indices.py` was blocked by a tool permission check. The Zenodo per-cell `plasticity_scores.tsv` and the definitions in the paper and its Figure 4 notebook were used instead. + +## Reproducing + +See [README.md](README.md). All randomness is seeded. The machine used had 48 cores, and the full pipeline takes about 2 hours. diff --git a/analysis/kptracer/bdmm/bdmm_fixedtree.xml b/analysis/kptracer/bdmm/bdmm_fixedtree.xml new file mode 100644 index 000000000..daa3f2a45 --- /dev/null +++ b/analysis/kptracer/bdmm/bdmm_fixedtree.xml @@ -0,0 +1,281 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +n123=1, +n185=2, +n186=1, +n130=0, +n66=1, +n175=0, +n176=0, +n159=0, +n160=0, +n137=0, +n33=0, +n34=1, +n149=0, +n150=0, +n47=0, +n96=2, +n100=2, +n99=2, +n197=2, +n198=2, +n177=0, +n178=1, +n191=1, +n192=1, +n139=2, +n181=0, +n182=0, +n131=2, +n106=1, +n127=2, +n121=2, +n122=2, +n135=2, +n136=2, +n143=2, +n144=2, +n195=1, +n196=1, +n147=1, +n148=1, +n45=2, +n46=1, +n37=2, +n165=2, +n110=1, +n171=3, +n172=3, +n69=1, +n71=1, +n193=2, +n194=2, +n187=1, +n188=1, +n102=0, +n103=1, +n104=1, +n53=2, +n85=1, +n86=1, +n88=0, +n125=0, +n73=0, +n151=3, +n152=3, +n77=2, +n78=1, +n120=1, +n133=0, +n134=1, +n141=1, +n142=1, +n183=0, +n184=0, +n117=1, +n118=1, +n60=1, +n91=1, +n92=0, +n161=1, +n157=0, +n158=0, +n167=0, +n168=0, +n63=2, +n113=0, +n179=1, +n180=1, +n108=1, +n163=0, +n164=0, +n84=0, +n89=1, +n90=1, +n153=1, +n154=1, +n155=0, +n156=1, +n170=0, +n189=1, +n190=1 + + + + + + + + 1.0 + 0.1 + 0.25 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/analysis/kptracer/bdmm/bdmm_summary.json b/analysis/kptracer/bdmm/bdmm_summary.json new file mode 100644 index 000000000..2fc16bc0e --- /dev/null +++ b/analysis/kptracer/bdmm/bdmm_summary.json @@ -0,0 +1,142 @@ +{ + "wall_seconds": 6445.654667854309, + "wall_seconds_per_chain": [ + 6375.539113044739, + 6445.653131723404, + 6400.728255271912, + 6434.687438011169 + ], + "seconds_per_million_samples": 2565.6607938051225, + "n_chains": 4, + "chain_length_per_chain": 2500000, + "log_every": 500, + "n_samples": 18004, + "n_samples_note": "post-burnin logged samples pooled across chains (10% burnin per chain); MCMC states per chain = 2.5e6", + "ess_min": 10432.411489106, + "ess_median": 11367.911747273869, + "ess_per_rate_pooled": [ + 13897.41471443144, + 10432.411489106, + 11084.62202576015, + 15691.542288148514, + 12320.547834141229, + 11485.265527526453, + 11758.14957153604, + 11250.557967021283, + 15326.36107722986, + 11094.206628933782, + 11035.891382524755, + 10621.721846149254 + ], + "ess_min_single_chain": 2440.0118219268766, + "ess_median_single_chain": 2897.048763724237, + "rhat_max": 1.0003411175377643, + "hpd_coverage": "12/12", + "rates_true": [ + [ + 0.0, + 0.44999999999999996, + 0.06, + 0.015 + ], + [ + 0.15000000000000002, + 0.0, + 0.30000000000000004, + 0.015 + ], + [ + 0.015, + 0.06, + 0.0, + 0.24 + ], + [ + 0.015, + 0.015, + 0.03, + 0.0 + ] + ], + "rates_median": [ + [ + 0.0, + 0.5103131085784036, + 0.07671978415025725, + 0.043024337623377054 + ], + [ + 0.28687421600480206, + 0.0, + 0.1979957382949188, + 0.0361916114083121 + ], + [ + 0.07348354619177175, + 0.13039641620359893, + 0.0, + 0.13211403694655444 + ], + [ + 0.21424332632389226, + 0.24762393891238374, + 0.2726695768798638, + 0.0 + ] + ], + "rates_lo": [ + [ + 0.0, + 0.1021538212235079, + 2.308699221686512e-05, + 8.110066720244403e-07 + ], + [ + 0.0421476778310851, + 0.0, + 0.0282492062121519, + 6.580723324512174e-06 + ], + [ + 4.293625169993541e-06, + 3.7391719475294177e-06, + 0.0, + 0.0003033097511343714 + ], + [ + 9.367528010825216e-06, + 3.5701304171177952e-06, + 8.712950332777805e-07, + 0.0 + ] + ], + "rates_hi": [ + [ + 0.0, + 0.9490902173754688, + 0.2951695575057029, + 0.1862328181528537 + ], + [ + 0.5915360122563034, + 0.0, + 0.413992040097056, + 0.1535975800165734 + ], + [ + 0.322119548035203, + 0.4841343296266506, + 0.0, + 0.3766452595871318 + ], + [ + 0.9030265762202508, + 1.0474542475169335, + 1.0592085015999635, + 0.0 + ] + ], + "beast_version": "2.7.7 (BEAST.base/BEAST.app 2.7.7, bundled JRE)", + "bdmm_prime_version": "2.7.2", + "notes": "Likelihood: BDMM-Prime BirthDeathMigrationDistribution (Kuehnert et al. 2016 ODE likelihood, integrated over unobserved type histories; not a mapped/typed-tree likelihood). Tree FIXED: loaded from tree100.nwk via TreeParser, not in the MCMC state, no tree operators. TreeParser adjustTipHeights=true was required: tips in the Newick differ by up to 3e-6 due to 6-decimal rounding, and BDMM-Prime otherwise does not treat them as rho-sampled (logP=-Inf). conditionOnRoot=true, processLength = root height (no stem). Model: 4 types (tip types from tip_types.tsv), single shared birth rate (LogNormal(0,1) prior), death=0, psi sampling rate=0, rho=1 at present, 12 migration rates (BDMM-Prime forward-time, row-major off-diagonal order from->to) with iid Exponential(mean 0.3) priors, root type frequencies (startTypePriorProbs) estimated with Dirichlet(1,1,1,1). Operators: single-element ScaleOperator on migration rates, scaleAll on migration rates, ScaleOperator on birth rate, DeltaExchange on root freqs. Run: 4 independent chains (seeds 1001-1004) run concurrently, 2.5e6 states each, log every 500, 10% burnin; BDMM-Prime internal parallelize=true (~1.8 cores/chain; host shared with another heavy job). wall_seconds = wall time for all 4 concurrent chains; a single chain alone is ~same speed (~2.2-2.8 ms/state). ESS via arviz 0.x bulk ESS (pooled over chains and for chain 1 alone). Mixing is good: single-chain ESS>2400 for all rates, so ESS>200 would need only ~2e5 states (~10 min single chain). Accuracy caveat: with 100 tips the posteriors are wide and prior-sensitive; all 12 true rates lie inside the 95% HPD, but rates out of rare types (esp. type 3) are essentially prior-dominated (median ~0.21-0.27 vs Exp(0.3) prior median 0.21). Installation: BEAST 2.7.7 Linux tarball (bundled JRE); packagemanager -add into ~/.beast failed (write error), so packages were installed with -dir /home/asm/data/bdmm/pkgs; the package manager also pulled BEAST.base/app 2.7.8 which conflicted with the bundled 2.7.7 and were removed. Run with beast -packagedir /home/asm/data/bdmm/pkgs." +} \ No newline at end of file diff --git a/analysis/kptracer/bdmm/tip_types.tsv b/analysis/kptracer/bdmm/tip_types.tsv new file mode 100644 index 000000000..89830336d --- /dev/null +++ b/analysis/kptracer/bdmm/tip_types.tsv @@ -0,0 +1,101 @@ +taxon type +n123 1 +n185 2 +n186 1 +n130 0 +n66 1 +n175 0 +n176 0 +n159 0 +n160 0 +n137 0 +n33 0 +n34 1 +n149 0 +n150 0 +n47 0 +n96 2 +n100 2 +n99 2 +n197 2 +n198 2 +n177 0 +n178 1 +n191 1 +n192 1 +n139 2 +n181 0 +n182 0 +n131 2 +n106 1 +n127 2 +n121 2 +n122 2 +n135 2 +n136 2 +n143 2 +n144 2 +n195 1 +n196 1 +n147 1 +n148 1 +n45 2 +n46 1 +n37 2 +n165 2 +n110 1 +n171 3 +n172 3 +n69 1 +n71 1 +n193 2 +n194 2 +n187 1 +n188 1 +n102 0 +n103 1 +n104 1 +n53 2 +n85 1 +n86 1 +n88 0 +n125 0 +n73 0 +n151 3 +n152 3 +n77 2 +n78 1 +n120 1 +n133 0 +n134 1 +n141 1 +n142 1 +n183 0 +n184 0 +n117 1 +n118 1 +n60 1 +n91 1 +n92 0 +n161 1 +n157 0 +n158 0 +n167 0 +n168 0 +n63 2 +n113 0 +n179 1 +n180 1 +n108 1 +n163 0 +n164 0 +n84 0 +n89 1 +n90 1 +n153 1 +n154 1 +n155 0 +n156 1 +n170 0 +n189 1 +n190 1 diff --git a/analysis/kptracer/bdmm/tree100.nwk b/analysis/kptracer/bdmm/tree100.nwk new file mode 100644 index 000000000..ad0ee486c --- /dev/null +++ b/analysis/kptracer/bdmm/tree100.nwk @@ -0,0 +1 @@ +((((n185:0.105435,n186:0.105435)n75:0.758506,((n175:0.169748,n176:0.169748)n145:0.112940,(n159:0.234216,n160:0.234216)n146:0.048472)n76:0.581254)n43:0.449058,(n123:0.450563,((n137:0.324692,(n149:0.272999,n150:0.272999)n138:0.051694)n129:0.099779,n130:0.424471)n124:0.026092)n44:0.862437)n1:1.968832,(((((n47:1.159471,(n181:0.117813,n182:0.117813)n48:1.041658)n23:1.098771,((n131:0.409898,(n165:0.217494,(n193:0.047186,n194:0.047186)n166:0.170309)n132:0.192403)n55:0.683481,(((n187:0.104629,n188:0.104629)n109:0.501520,n110:0.606149)n105:0.017891,n106:0.624040)n56:0.469339)n24:1.164863)n15:0.438788,(((n127:0.431499,(n171:0.173877,n172:0.173877)n128:0.257622)n95:0.332616,n96:0.764115)n79:0.093977,(n100:0.734049,n99:0.734049)n80:0.124044)n16:1.838938)n5:0.300383,((n33:1.783769,n34:1.783769)n21:0.587406,(((n121:0.490003,n122:0.490003)n111:0.102243,(n135:0.369630,n136:0.369630)n112:0.222616)n97:0.169138,(n197:0.012586,n198:0.012586)n98:0.748797)n22:1.609791)n6:0.626239)n3:0.015266,((((n177:0.135231,n178:0.135231)n173:0.037077,(n191:0.052557,n192:0.052557)n174:0.119751)n65:0.745767,n66:0.918074)n7:2.050193,(((n139:0.323601,(n143:0.288174,n144:0.288174)n140:0.035427)n41:1.038820,((n195:0.015571,n196:0.015571)n115:0.514336,(n147:0.281240,n148:0.281240)n116:0.248668)n42:0.832513)n10:1.591565,((((n69:0.909870,((n125:0.436570,(n183:0.108761,n184:0.108761)n126:0.327809)n101:0.212951,n102:0.649521)n70:0.260349)n19:1.594987,((n103:0.645152,n104:0.645152)n49:0.464789,(n53:1.095520,(n73:0.887923,((n157:0.235615,n158:0.235615)n81:0.607617,(n167:0.214877,n168:0.214877)n82:0.628355)n74:0.044692)n54:0.207596)n50:0.014421)n20:1.394916)n17:0.101622,((n71:0.903043,(n85:0.829271,n86:0.829271)n72:0.073772)n25:1.128496,(((n151:0.267928,n152:0.267928)n31:1.569555,(n77:0.859500,n78:0.859500)n32:0.977983)n27:0.069964,(((n117:0.510409,n118:0.510409)n35:1.142729,((n63:0.927585,((n153:0.264503,n154:0.264503)n107:0.346663,n108:0.611166)n64:0.316418)n59:0.114692,n60:1.042276)n36:0.610861)n29:0.219068,((n91:0.777811,n92:0.777811)n57:0.302458,((n113:0.531419,(n163:0.218871,n164:0.218871)n114:0.312548)n61:0.413352,((((n155:0.236850,n156:0.236850)n93:0.527531,((n189:0.086787,n190:0.086787)n169:0.097964,n170:0.184751)n94:0.579630)n83:0.066969,n84:0.831350)n67:0.078867,(n89:0.779814,n90:0.779814)n68:0.130402)n62:0.034554)n58:0.135498)n30:0.791937)n28:0.035241)n26:0.124093)n18:0.574939)n11:0.316059,((n45:1.179748,n46:1.179748)n13:1.605255,(n37:1.583666,((((n161:0.221276,(n179:0.121742,n180:0.121742)n162:0.099534)n119:0.287553,n120:0.508829)n51:0.592224,(n133:0.382097,n134:0.382097)n52:0.718956)n39:0.284872,((n141:0.313129,n142:0.313129)n87:0.509341,n88:0.822470)n40:0.563455)n38:0.197741)n14:1.201337)n12:0.137534)n9:0.031449)n8:0.014281)n4:0.044412)n2:0.269154)n0:0.000000; diff --git a/analysis/kptracer/bdmm/truth.json b/analysis/kptracer/bdmm/truth.json new file mode 100644 index 000000000..f858d914e --- /dev/null +++ b/analysis/kptracer/bdmm/truth.json @@ -0,0 +1 @@ +{"Q": [[-0.5249999999999999, 0.44999999999999996, 0.06, 0.015], [0.15000000000000002, -0.4650000000000001, 0.30000000000000004, 0.015], [0.015, 0.06, -0.31500000000000006, 0.24], [0.015, 0.015, 0.03, -0.06]], "pi": [0.85, 0.1, 0.04, 0.01], "birth_rate": 1.0, "all_states": {"n0": 0, "n1": 0, "n2": 0, "n43": 0, "n44": 0, "n3": 0, "n4": 0, "n75": 1, "n76": 0, "n123": 1, "n124": 0, "n5": 0, "n6": 0, "n7": 1, "n8": 0, "n185": 2, "n186": 1, "n145": 0, "n146": 0, "n129": 0, "n130": 0, "n15": 0, "n16": 2, "n21": 0, "n22": 2, "n65": 1, "n66": 1, "n10": 1, "n9": 0, "n175": 0, "n176": 0, "n159": 0, "n160": 0, "n137": 0, "n138": 0, "n23": 0, "n24": 1, "n79": 2, "n80": 2, "n33": 0, "n34": 1, "n97": 2, "n98": 2, "n173": 1, "n174": 1, "n41": 2, "n42": 1, "n11": 1, "n12": 0, "n149": 0, "n150": 0, "n47": 0, "n48": 0, "n55": 2, "n56": 1, "n95": 2, "n96": 2, "n100": 2, "n99": 2, "n111": 2, "n112": 2, "n197": 2, "n198": 2, "n177": 0, "n178": 1, "n191": 1, "n192": 1, "n139": 2, "n140": 2, "n115": 1, "n116": 1, "n17": 1, "n18": 1, "n13": 1, "n14": 0, "n181": 0, "n182": 0, "n131": 2, "n132": 2, "n105": 1, "n106": 1, "n127": 2, "n128": 3, "n121": 2, "n122": 2, "n135": 2, "n136": 2, "n143": 2, "n144": 2, "n195": 1, "n196": 1, "n147": 1, "n148": 1, "n19": 0, "n20": 0, "n25": 1, "n26": 1, "n45": 2, "n46": 1, "n37": 2, "n38": 1, "n165": 2, "n166": 2, "n109": 1, "n110": 1, "n171": 3, "n172": 3, "n69": 1, "n70": 0, "n49": 1, "n50": 0, "n71": 1, "n72": 1, "n27": 1, "n28": 1, "n39": 1, "n40": 1, "n193": 2, "n194": 2, "n187": 1, "n188": 1, "n101": 0, "n102": 0, "n103": 1, "n104": 1, "n53": 2, "n54": 0, "n85": 1, "n86": 1, "n31": 3, "n32": 1, "n29": 1, "n30": 1, "n51": 1, "n52": 1, "n87": 1, "n88": 0, "n125": 0, "n126": 0, "n73": 0, "n74": 0, "n151": 3, "n152": 3, "n77": 2, "n78": 1, "n35": 1, "n36": 1, "n57": 1, "n58": 1, "n119": 1, "n120": 1, "n133": 0, "n134": 1, "n141": 1, "n142": 1, "n183": 0, "n184": 0, "n81": 0, "n82": 0, "n117": 1, "n118": 1, "n59": 1, "n60": 1, "n91": 1, "n92": 0, "n61": 0, "n62": 1, "n161": 1, "n162": 1, "n157": 0, "n158": 0, "n167": 0, "n168": 0, "n63": 2, "n64": 1, "n113": 0, "n114": 0, "n67": 0, "n68": 1, "n179": 1, "n180": 1, "n107": 1, "n108": 1, "n163": 0, "n164": 0, "n83": 0, "n84": 0, "n89": 1, "n90": 1, "n153": 1, "n154": 1, "n93": 0, "n94": 1, "n155": 0, "n156": 1, "n169": 1, "n170": 0, "n189": 1, "n190": 1}} \ No newline at end of file diff --git a/analysis/kptracer/bdmm_comparison.py b/analysis/kptracer/bdmm_comparison.py new file mode 100644 index 000000000..176737192 --- /dev/null +++ b/analysis/kptracer/bdmm_comparison.py @@ -0,0 +1,94 @@ +"""tHMM on the 100-leaf tree used for the BDMM-Prime comparison, with the same tip types as data. + +Emissions are fixed near-indicators of the observed type, so the tHMM sees exactly what BDMM-Prime +sees (tree, branch lengths, tip types). Intervals come from a parametric bootstrap. Writes +``results/bdmm_thmm.json``. Inputs are in ``bdmm/`` (regenerate them with ``--make-tree``), together +with the BDMM-Prime XML and its posterior summary (BEAST 2.7.7, BDMM-Prime 2.7.2). +""" + +import json +import sys +import time +from pathlib import Path + +import numpy as np +import pandas as pd + +from lineage.phylo_sim import simulate_states, yule_tree +from lineage.phyloHMM import PhyloHMM +from lineage.states.StateDistributionGaussian import StateDistribution as Gaussian +from lineage.tree_io import graph_to_phylotree, load_tree, to_lineage + +BDMM = Path(__file__).parent / "bdmm" +RES = Path(__file__).parent / "results" +K = 4 + + +def fit(pt, types, rng): + onehot = np.eye(K)[types] + E = [Gaussian(mean=np.eye(K)[k], var=np.full(K, 0.01)) for k in range(K)] + lin = to_lineage(pt, pt.leaf_names, onehot, E, branch_lengths=pt.branch_lengths) + return PhyloHMM([lin], K, rng=rng, fix_E=E).fit(init=False, tol=1e-7, max_iter=2000) + + +def make_tree(): + """The 100-leaf Yule tree and tip types given to both methods (rates 3x the simulation study's).""" + from simulation_study import PI_TRUE, Q_TRUE + + rng = np.random.default_rng(11) + Q = Q_TRUE * 3 + pt = graph_to_phylotree(yule_tree(100, rng=rng)) + s = simulate_states(pt, Q, PI_TRUE, rng) + + def nwk(i): + ch = pt.tree.indices[pt.tree.indptr[i] : pt.tree.indptr[i + 1]] + inner = "(" + ",".join(nwk(c) for c in ch) + ")" if len(ch) else "" + return f"{inner}{pt.names[i]}:{pt.branch_lengths[i]:.6f}" + + BDMM.mkdir(exist_ok=True) + (BDMM / "tree100.nwk").write_text(nwk(0) + ";\n") + tips = [f"{n}\t{st}" for n, st, leaf in zip(pt.names, s, pt.is_leaf, strict=True) if leaf] + (BDMM / "tip_types.tsv").write_text("taxon\ttype\n" + "\n".join(tips) + "\n") + truth = {"Q": Q.tolist(), "pi": PI_TRUE.tolist(), "birth_rate": 1.0} + truth["all_states"] = dict(zip(map(str, pt.names), map(int, s), strict=True)) + (BDMM / "truth.json").write_text(json.dumps(truth)) + + +if __name__ == "__main__": + if "--make-tree" in sys.argv: + make_tree() + sys.exit() + truth = json.loads((BDMM / "truth.json").read_text()) + pt = load_tree(BDMM / "tree100.nwk", collapse_unifurcations=False) + tips = pd.read_csv(BDMM / "tip_types.tsv", sep="\t", index_col=0)["type"] + types = tips.loc[list(pt.leaf_names)].values.astype(int) + + t0 = time.perf_counter() + m = fit(pt, types, 0) + wall = time.perf_counter() - t0 + + rng = np.random.default_rng(1) + boots = [] + t0 = time.perf_counter() + for _ in range(200): + s = simulate_states(pt, m.Q, m.pi, rng) + boots.append(fit(pt, s[pt.is_leaf], rng).Q) + boot_wall = time.perf_counter() - t0 + boots = np.array(boots) + Qtrue = np.array(truth["Q"]) + off = ~np.eye(K, dtype=bool) + lo, hi = np.percentile(boots, 2.5, axis=0), np.percentile(boots, 97.5, axis=0) + out = { + "fit_seconds": wall, + "bootstrap_seconds_200": boot_wall, + "rates_true": Qtrue.tolist(), + "rates_mle": m.Q.tolist(), + "rates_lo": lo.tolist(), + "rates_hi": hi.tolist(), + "coverage": float(np.mean((Qtrue[off] >= lo[off]) & (Qtrue[off] <= hi[off]))), + "median_abs_log_ratio": float(np.median(np.abs(np.log(m.Q[off] / np.maximum(Qtrue[off], 1e-6))))), + } + (RES / "bdmm_thmm.json").write_text(json.dumps(out, indent=1)) + print(json.dumps({k: v for k, v in out.items() if not k.startswith("rates")}, indent=1)) + print(np.round(m.Q, 3)) + print(np.round(Qtrue, 3)) diff --git a/analysis/kptracer/figures/model_selection.png b/analysis/kptracer/figures/model_selection.png new file mode 100644 index 000000000..8ed18e9f9 Binary files /dev/null and b/analysis/kptracer/figures/model_selection.png differ diff --git a/analysis/kptracer/figures/scalability.png b/analysis/kptracer/figures/scalability.png new file mode 100644 index 000000000..784c7cf98 Binary files /dev/null and b/analysis/kptracer/figures/scalability.png differ diff --git a/analysis/kptracer/figures/simulation_Q_error.png b/analysis/kptracer/figures/simulation_Q_error.png new file mode 100644 index 000000000..5f4848b80 Binary files /dev/null and b/analysis/kptracer/figures/simulation_Q_error.png differ diff --git a/analysis/kptracer/figures/simulation_accuracy.png b/analysis/kptracer/figures/simulation_accuracy.png new file mode 100644 index 000000000..4e877ecba Binary files /dev/null and b/analysis/kptracer/figures/simulation_accuracy.png differ diff --git a/analysis/kptracer/figures/transitions_anchored.png b/analysis/kptracer/figures/transitions_anchored.png new file mode 100644 index 000000000..e140d6311 Binary files /dev/null and b/analysis/kptracer/figures/transitions_anchored.png differ diff --git a/analysis/kptracer/figures/tumor_benchmarks_anchored.png b/analysis/kptracer/figures/tumor_benchmarks_anchored.png new file mode 100644 index 000000000..71f8c9db3 Binary files /dev/null and b/analysis/kptracer/figures/tumor_benchmarks_anchored.png differ diff --git a/analysis/kptracer/figures/unsup_vs_clusters.png b/analysis/kptracer/figures/unsup_vs_clusters.png new file mode 100644 index 000000000..a2086c481 Binary files /dev/null and b/analysis/kptracer/figures/unsup_vs_clusters.png differ diff --git a/analysis/kptracer/kptracer_analysis.py b/analysis/kptracer/kptracer_analysis.py new file mode 100644 index 000000000..6134cdb7b --- /dev/null +++ b/analysis/kptracer/kptracer_analysis.py @@ -0,0 +1,460 @@ +"""Fit the branch-length tHMM to KP-Tracer tumors and compare it with parsimony, PATH and published scores. + +Stages (run in order; each writes into ``results/``):: + + uv run python analysis/kptracer/kptracer_analysis.py select # K and branch-length model selection + uv run python analysis/kptracer/kptracer_analysis.py fit # final unsupervised + cluster-anchored fits + uv run python analysis/kptracer/kptracer_analysis.py compare # per-tumor benchmarks + +Requires ``results/kptracer_inputs.pkl`` from ``prepare_data.py``. +""" + +import os + +os.environ.setdefault("OMP_NUM_THREADS", "1") +os.environ.setdefault("OPENBLAS_NUM_THREADS", "1") + +import pickle +import sys +from concurrent.futures import ProcessPoolExecutor +from pathlib import Path + +import numpy as np +import pandas as pd +from scipy import stats + +from lineage.phylo_stats import ( + leaf_node_distance, + phylogenetic_correlation, + phylogenetic_correlation_z, + small_parsimony, +) +from lineage.phyloHMM import PhyloHMM, fit_best, mask_leaves +from lineage.states.StateDistributionGaussian import StateDistribution as Gaussian +from lineage.tree_io import to_lineage + +RES = Path(__file__).parent / "results" +MAIN_DELTA = 0.5 + + +def load_records(exclude=()): + with open(RES / "kptracer_inputs.pkl", "rb") as f: + recs = pickle.load(f) + return [r for r in recs if r["tumor"] not in exclude] + + +def branch_lengths(rec, model: str, delta: float) -> np.ndarray: + """``mut``: mutations on the edge + delta. ``unit``: every edge has length 1.""" + if model == "unit": + bl = np.ones(len(rec["n_mut"])) + else: + bl = rec["n_mut"] + delta + bl[0] = 0.0 + return bl + + +def build(recs, model="mut", delta=MAIN_DELTA, emb="scvi"): + D = recs[0][emb].shape[1] + return [ + to_lineage( + r["tree"], r["leaf_names"], r[emb], [Gaussian(dim=D)], branch_lengths=branch_lengths(r, model, delta) + ) + for r in recs + ] + + +# ---------------------------------------------------------------------------- stage 1: selection + + +def select_job(args): + K, rep, model, delta, exclude = args + recs = load_records(exclude) + X = build(recs, model, delta) + Xm = mask_leaves(X, 0.2, rng=1000 + rep) + m = fit_best(Xm, K, n_init=2, rng=rep, tol=1e-4, max_iter=200) + m_full = fit_best(X, K, n_init=2, rng=rep, tol=1e-4, max_iter=200) if rep == 0 else None + row = { + "K": K, + "rep": rep, + "bl_model": model, + "delta": delta, + "excluded": ",".join(exclude), + "heldout_ll": m.heldout_loglik(Xm, X), + "heldout_ll_no_tree": m.heldout_loglik(Xm, X, lineage=False), + "train_ll": m.LL, + } + if m_full is not None: + row.update({"full_ll": m_full.LL, "BIC": m_full.BIC(), "n_params": m_full.num_parameters()}) + print(row, flush=True) + return row + + +def stage_select(): + jobs = [(K, rep, "mut", MAIN_DELTA, ()) for K in range(2, 15) for rep in range(3)] + # Branch-length sensitivity at a few K + for K in (6, 8, 10): + for model, delta in [("mut", 0.1), ("mut", 0.25), ("mut", 1.0), ("mut", 2.0), ("unit", 0.0)]: + jobs += [(K, rep, model, delta, ()) for rep in range(3)] + with ProcessPoolExecutor(46) as ex: + rows = list(ex.map(select_job, jobs)) + pd.DataFrame(rows).to_csv(RES / "model_selection.csv", index=False) + + +# ---------------------------------------------------------------------------- stage 2: fits + + +def fit_job(args): + name, K, exclude, anchored, delta, model, emb = args + recs = load_records(exclude) + X = build(recs, model, delta, emb) + if anchored: + # One state per published Leiden cluster present after filtering, with emissions started at + # the cluster means; EM is then free to move them. + labels = np.concatenate([r["leiden"] for r in recs]) + uniq = np.unique(labels) + obs = np.vstack([r[emb] for r in recs]) + E = [] + for c in uniq: + g = Gaussian(dim=obs.shape[1]) + g.estimator(obs, (labels == c).astype(float)) + E.append(g) + # By default the emissions stay fixed at the cluster-conditional Gaussians, so every state keeps + # its published identity; with free emissions EM relabels states (see the report). + m = PhyloHMM(X, len(uniq), rng=0, fix_E=None if anchored == "free" else E) + if anchored == "free": + m.E = E + m.fit(init=False, tol=1e-5, max_iter=400) + state_labels = uniq + else: + m = fit_best(X, K, n_init=5, rng=0, tol=1e-5, max_iter=400) + state_labels = None + out = { + "name": name, + "tumors": [r["tumor"] for r in recs], + "Q": m.Q, + "pi": m.pi, + "means": np.array([e.mean for e in m.E]), + "vars": np.array([e.var for e in m.E]), + "LL": m.LL, + "BIC": m.BIC(), + "gammas": [p.gamma for p in m.posteriors], + "summary": m.switch_summary(), + "state_labels": state_labels, + "iters": len(m.LL_trace), + "delta": delta, + "bl_model": model, + "emb": emb, + "fixed_E": bool(anchored) and anchored != "free", + } + with open(RES / f"fit_{name}.pkl", "wb") as f: + pickle.dump(out, f) + print(name, m.LL, len(m.LL_trace), flush=True) + return name + + +def stage_fit(K: int, anchored_only: bool = False): + jobs = [ + ("anchored", None, (), True, MAIN_DELTA, "mut", "scvi"), + ("anchored_no3724", None, ("3724_NT_T1",), True, MAIN_DELTA, "mut", "scvi"), + ("anchored_delta0.1", None, (), True, 0.1, "mut", "scvi"), + ("anchored_delta2", None, (), True, 2.0, "mut", "scvi"), + ("anchored_unit", None, (), True, 0.0, "unit", "scvi"), + ("anchored_freeE", None, (), "free", MAIN_DELTA, "mut", "scvi"), + ] + if not anchored_only: + jobs.append(("unsup", K, (), False, MAIN_DELTA, "mut", "scvi")) + jobs.append(("unsup_no3724", K, ("3724_NT_T1",), False, MAIN_DELTA, "mut", "scvi")) + jobs.append(("unsup_pca", K, (), False, MAIN_DELTA, "mut", "pca")) + with ProcessPoolExecutor(len(jobs)) as ex: + list(ex.map(fit_job, jobs)) + + +# ---------------------------------------------------------------------------- stage 2b: bootstrap + + +def bootstrap_job(args): + b, name = args + with open(RES / f"fit_{name}.pkl", "rb") as f: + fit = pickle.load(f) + recs = [r for r in load_records() if r["tumor"] in fit["tumors"]] + rng = np.random.default_rng(b) + pick = rng.integers(len(recs), size=len(recs)) + X = build([recs[i] for i in pick], fit["bl_model"], fit["delta"], fit["emb"]) + E = [Gaussian(mean=mu, var=v) for mu, v in zip(fit["means"], fit["vars"], strict=True)] + m = PhyloHMM(X, len(fit["pi"]), rng=b, fix_E=E if fit.get("fixed_E") else None) + m.E = [Gaussian(mean=mu, var=v) for mu, v in zip(fit["means"], fit["vars"], strict=True)] + m.Q, m.pi = fit["Q"].copy(), fit["pi"].copy() + m.fit(init=False, tol=1e-5, max_iter=400) + summ = m.switch_summary() + joint = sum(s["edge_joint"] for s in summ) + jumps = sum(s["jumps"] for s in summ) + return { + "b": b, + "Q": m.Q, + "pi": m.pi, + "stay": np.diag(joint) / joint.sum(axis=1), + "jumps": jumps, + "root": np.mean([p.gamma[0] for p in m.posteriors], axis=0), + } + + +def stage_bootstrap(name="anchored", B=48): + with ProcessPoolExecutor(min(B, 46)) as ex: + res = list(ex.map(bootstrap_job, [(b, name) for b in range(B)])) + with open(RES / f"bootstrap_{name}.pkl", "wb") as f: + pickle.dump(res, f) + + +# ---------------------------------------------------------------------------- stage 3: compare + + +def tumor_benchmarks(fit, recs, max_path_leaves=4000, rng=0): + """Per-tumor tHMM plasticity vs Fitch parsimony, PATH auto-correlation, and published scPlasticity.""" + rng = np.random.default_rng(rng) + rows, cell_rows = [], [] + for r, gam, summ in zip(recs, fit["gammas"], fit["summary"], strict=True): + pt = r["tree"] + leaf = pt.is_leaf + uniq = np.unique(r["leiden"]) + lab_small = np.searchsorted(uniq, r["leiden"]) + lab = np.full(len(pt.names), -1) + lab[leaf] = lab_small + score, _, _ = small_parsimony(pt, lab, len(uniq)) + n_edges = len(pt.names) - 1 + + # PATH-style Moran's I of the Leiden labels, averaged over states weighted by frequency + leaves = np.nonzero(leaf)[0] + sel = np.arange(leaves.size) + if leaves.size > max_path_leaves: + sel = np.sort(rng.choice(leaves.size, max_path_leaves, replace=False)) + dist = leaf_node_distance(pt, leaves[sel]) + C, Cz = phylogenetic_correlation_z(lab_small[sel], dist, len(uniq), n_perm=100, rng=rng) + freq = np.bincount(lab_small[sel], minlength=len(uniq)) / sel.size + path_auto = float(np.sum(np.diag(C) * freq)) + path_auto_z = float(np.sum(np.diag(Cz) * freq)) + + # Same Moran's I on tHMM MAP states, to separate the state definition from the method + map_state = np.argmax(gam[leaf], axis=1) + C2 = phylogenetic_correlation(map_state[sel], dist, gam.shape[1]) + f2 = np.bincount(map_state[sel], minlength=gam.shape[1]) / sel.size + path_auto_thmm = float(np.nansum(np.diag(C2) * f2)) + + # Per-cell tHMM plasticity: mean edge-change probability along the root-to-leaf path + par = pt.parents + pe = np.zeros(len(pt.names)) + pe[summ_children(pt)] = summ["per_edge_change"] + cum = np.zeros(len(pt.names)) + nedge = np.zeros(len(pt.names)) + for i in range(1, len(pt.names)): + cum[i] = cum[par[i]] + pe[i] + nedge[i] = nedge[par[i]] + 1 + cell_score = cum[leaf] / nedge[leaf] + pub = r["published_scPlasticity"] + ok = np.isfinite(pub) + cell_rows.append( + pd.DataFrame( + {"tumor": r["tumor"], "cell": r["leaf_names"], "thmm_cell_plasticity": cell_score, "published": pub} + ) + ) + rows.append( + { + "tumor": r["tumor"], + "n_cells": int(leaf.sum()), + "n_edges": n_edges, + "n_clusters": len(uniq), + "fitch_changes": score, + "fitch_plasticity": score / n_edges, + "thmm_changes": summ["expected_changes"], + "thmm_plasticity": summ["plasticity"], + "thmm_jumps": float(summ["jumps"].sum()), + "path_autocorr_leiden": path_auto, + "path_autocorr_z_leiden": path_auto_z, + "path_autocorr_thmm": path_auto_thmm, + "published_mean_scPlasticity": float(np.nanmean(pub)) if ok.any() else np.nan, + "cell_spearman_vs_published": stats.spearmanr(cell_score[ok], pub[ok])[0] if ok.sum() > 10 else np.nan, + } + ) + return pd.DataFrame(rows), pd.concat(cell_rows) + + +def summ_children(pt): + """Children in CSR edge order, which is the order of ``per_edge_change``.""" + return pt.tree.indices + + +def pooled_transition_tables(fit, recs): + """Pooled k -> l change counts: tHMM expected edge changes and CTMC jumps (anchored states), and Fitch.""" + labels = fit["state_labels"] + K = len(labels) + thmm = sum(s["edge_changes"] for s in fit["summary"]) + jumps = sum(s["jumps"] for s in fit["summary"]) + fitch = np.zeros((K, K)) + for r in recs: + pt = r["tree"] + lab = np.full(len(pt.names), -1) + lab[pt.is_leaf] = np.searchsorted(labels, r["leiden"]) + _, _, counts = small_parsimony(pt, lab, K) + fitch += counts + return thmm, jumps, fitch + + +def state_table(fit, recs, jumps): + """Per-state stability from the anchored fit, next to Fitch and the published per-cell scores.""" + labels = fit["state_labels"] + K = len(labels) + names = {} + for r in recs: + names.update(dict(zip(r["leiden"].tolist(), r["cluster_name"].tolist(), strict=True))) + joint = sum(s["edge_joint"] for s in fit["summary"]) # pooled P(parent = k, child = l) + fitch_edges = np.zeros((K, K)) + root = np.zeros(K) + n_cells = np.zeros(K) + pub: dict[int, list[float]] = {k: [] for k in range(K)} + for r, gam in zip(recs, fit["gammas"], strict=True): + pt = r["tree"] + leaf_lab = np.searchsorted(labels, r["leiden"]) + lab = np.full(len(pt.names), -1) + lab[pt.is_leaf] = leaf_lab + _, fl, _ = small_parsimony(pt, lab, K) + np.add.at(fitch_edges, (fl[pt.parents[1:]], fl[1:]), 1) + root += gam[0] + n_cells += np.bincount(leaf_lab, minlength=K) + for k in range(K): + v = r["published_scPlasticity"][leaf_lab == k] + v = v[np.isfinite(v)] + if v.size: + pub[k].append(float(v.mean())) + Q = fit["Q"] + return pd.DataFrame( + { + "leiden": labels, + "name": [names.get(int(k), str(k)) for k in labels], + "n_cells": n_cells.astype(int), + "exit_rate": -np.diag(Q), + "thmm_stay_prob": np.diag(joint) / np.maximum(joint.sum(axis=1), 1e-12), + "fitch_stay_prob": np.diag(fitch_edges) / np.maximum(fitch_edges.sum(axis=1), 1), + # Median over tumors of the per-tumor mean, as in Yang et al. (higher = more plastic) + "published_scPlasticity": [np.median(pub[k]) if pub[k] else np.nan for k in range(K)], + "root_posterior": root / len(recs), + "stationary": _stationary(Q), + "net_outflow_jumps": jumps.sum(axis=1) - jumps.sum(axis=0), + } + ) + + +def _stationary(Q): + from lineage.ctmc import stationary + + return stationary(Q) + + +def bootstrap_summary(name="anchored"): + """95% tumor-bootstrap intervals for per-state stability and for the direction of the largest flows.""" + path = RES / f"bootstrap_{name}.pkl" + if not path.exists(): + return + with open(path, "rb") as f: + boot = pickle.load(f) + st = pd.read_csv(RES / f"state_table_{name}.csv") + stay = np.array([b["stay"] for b in boot]) + exit_rate = np.array([-np.diag(b["Q"]) for b in boot]) + root = np.array([b["root"] for b in boot]) + for key, arr in (("stay", stay), ("exit_rate", exit_rate), ("root", root)): + st[f"{key}_lo"] = np.nanpercentile(arr, 2.5, axis=0) + st[f"{key}_hi"] = np.nanpercentile(arr, 97.5, axis=0) + st["stay_rank_median"] = np.median(np.argsort(np.argsort(-stay, axis=1), axis=1), axis=0) + 1 + st.to_csv(RES / f"state_table_{name}.csv", index=False) + + J = pd.read_csv(RES / f"transitions_jumps_{name}.csv", index_col=0).values + jumps = np.array([b["jumps"] for b in boot]) + names = st["name"].tolist() + rows = [] + for i, j in zip(*np.triu_indices(len(names), 1), strict=True): + total = J[i, j] + J[j, i] + if total < 50: + continue + net = jumps[:, i, j] - jumps[:, j, i] + frac = jumps[:, i, j] / np.maximum(jumps[:, i, j] + jumps[:, j, i], 1e-12) + a, b = (i, j) if J[i, j] >= J[j, i] else (j, i) + rows.append( + { + "from": names[a], + "to": names[b], + "jumps_fwd": J[a, b], + "jumps_rev": J[b, a], + "fwd_fraction": J[a, b] / (J[a, b] + J[b, a]), + "fwd_fraction_lo": np.percentile(frac if a == i else 1 - frac, 2.5), + "fwd_fraction_hi": np.percentile(frac if a == i else 1 - frac, 97.5), + "boot_same_direction": float(np.mean(np.sign(net) == np.sign(J[i, j] - J[j, i]))), + } + ) + pd.DataFrame(rows).sort_values("jumps_fwd", ascending=False).to_csv(RES / f"flows_{name}.csv", index=False) + + +def stage_compare(): + out = {} + for name in ( + "anchored", + "unsup", + "anchored_no3724", + "anchored_delta0.1", + "anchored_delta2", + "anchored_unit", + "anchored_freeE", + ): + path = RES / f"fit_{name}.pkl" + if not path.exists(): + continue + with open(path, "rb") as f: + fit = pickle.load(f) + recs = [r for r in load_records() if r["tumor"] in fit["tumors"]] + tb, cells = tumor_benchmarks(fit, recs) + tb.to_csv(RES / f"tumor_benchmarks_{name}.csv", index=False) + if name in ("anchored", "unsup"): + cells.to_csv(RES / f"cell_plasticity_{name}.csv.gz", index=False) + pairs = [ + ("thmm_plasticity", "fitch_plasticity"), + ("thmm_plasticity", "published_mean_scPlasticity"), + ("fitch_plasticity", "published_mean_scPlasticity"), + ("thmm_plasticity", "path_autocorr_leiden"), + ("fitch_plasticity", "path_autocorr_leiden"), + ("thmm_plasticity", "path_autocorr_thmm"), + ("thmm_plasticity", "path_autocorr_z_leiden"), + ("fitch_plasticity", "path_autocorr_z_leiden"), + ] + out[name] = {f"{a}~{b}": stats.spearmanr(tb[a], tb[b])[0] for a, b in pairs} + allc = cells.dropna() + out[name]["cell_pooled_spearman"] = stats.spearmanr(allc.thmm_cell_plasticity, allc.published)[0] + out[name]["cell_median_within_tumor_spearman"] = float(np.nanmedian(tb.cell_spearman_vs_published)) + if fit["state_labels"] is not None: + thmm, jumps, fitch = pooled_transition_tables(fit, recs) + lab = [str(x) for x in fit["state_labels"]] + pd.DataFrame(thmm, index=lab, columns=lab).to_csv(RES / f"transitions_thmm_{name}.csv") + pd.DataFrame(jumps, index=lab, columns=lab).to_csv(RES / f"transitions_jumps_{name}.csv") + pd.DataFrame(fitch, index=lab, columns=lab).to_csv(RES / f"transitions_fitch_{name}.csv") + off = ~np.eye(len(lab), dtype=bool) + out[name]["transition_matrix_spearman_thmm_vs_fitch"] = stats.spearmanr(thmm[off], fitch[off])[0] + out[name]["total_changes_thmm"] = float(thmm.sum()) + out[name]["total_changes_fitch"] = float(fitch.sum()) + out[name]["total_jumps_ctmc"] = float(jumps.sum()) + st = state_table(fit, recs, jumps) + st.to_csv(RES / f"state_table_{name}.csv", index=False) + for a, b in [("thmm_stay_prob", "fitch_stay_prob"), ("thmm_stay_prob", "published_scPlasticity")]: + out[name][f"state_{a}~{b}"] = stats.spearmanr(st[a], st[b])[0] + pd.DataFrame(out).T.to_csv(RES / "benchmark_correlations.csv") + bootstrap_summary("anchored") + print(pd.DataFrame(out).T.round(3).to_string()) + + +if __name__ == "__main__": + stage = sys.argv[1] + if stage == "select": + stage_select() + elif stage == "fit": + stage_fit(int(sys.argv[2])) + elif stage == "fit_anchored": + stage_fit(0, anchored_only=True) + elif stage == "bootstrap": + stage_bootstrap(*sys.argv[2:3]) + elif stage == "compare": + stage_compare() diff --git a/analysis/kptracer/make_figures.py b/analysis/kptracer/make_figures.py new file mode 100644 index 000000000..40ac7604d --- /dev/null +++ b/analysis/kptracer/make_figures.py @@ -0,0 +1,260 @@ +"""Figures for the KP-Tracer report. Reads ``results/`` and writes PNGs to ``figures/``.""" + +import pickle +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np +import pandas as pd + +HERE = Path(__file__).parent +RES = HERE / "results" +FIG = HERE / "figures" + +# Categorical slots in fixed order (reference palette, light mode) +C = ["#2a78d6", "#eb6834", "#1baf7a", "#eda100", "#e87ba4", "#008300", "#4a3aa7", "#e34948"] +INK, INK2, GRID = "#0b0b0b", "#52514e", "#e4e3df" +METHODS = { + "tHMM-ctmc": ("tHMM, exp(Qt) per edge", C[0]), + "tHMM-discrete": ("tHMM, one T per edge", C[1]), + "parsimony-kmeans": ("k-means + parsimony", C[2]), + "parsimony-oracle": ("true labels + parsimony", C[3]), +} + +plt.rcParams.update( + { + "font.size": 9, + "axes.edgecolor": INK2, + "axes.labelcolor": INK, + "xtick.color": INK2, + "ytick.color": INK2, + "axes.spines.top": False, + "axes.spines.right": False, + "axes.grid": True, + "grid.color": GRID, + "grid.linewidth": 0.6, + "legend.frameon": False, + "figure.dpi": 150, + } +) + + +def fig_simulation(): + df = pd.read_csv(RES / "simulation.csv") + metrics = [ + ("leaf_acc", "Leaf state accuracy"), + ("anc_acc", "Ancestral state accuracy"), + ("changes_ratio", "Estimated / true state changes"), + ] + fig, axes = plt.subplots(2, 3, figsize=(10, 5.6), sharex=True) + for row, sep in enumerate([1.0, 0.5]): + sub = df[df.sep == sep] + for col, (m, title) in enumerate(metrics): + ax = axes[row, col] + for key, (label, color) in METHODS.items(): + g = sub[sub.method == key].groupby("frac")[m] + mu, sd = g.mean(), g.std() + ax.errorbar( + mu.index, mu.values, yerr=sd.values, color=color, lw=2, marker="o", ms=4, capsize=2, label=label + ) + ax.set_xscale("log") + ax.set_xticks([0.1, 0.25, 0.5, 1.0], ["10%", "25%", "50%", "100%"]) + if m == "changes_ratio": + ax.axhline(1.0, color=INK2, lw=0.8, ls="--") + ax.set_title(f"{title}\n{'well separated' if sep == 1.0 else 'overlapping'} states", fontsize=9) + if row == 1: + ax.set_xlabel("Leaves sampled") + handles, labels = axes[0, 0].get_legend_handles_labels() + fig.legend(handles, labels, loc="lower center", ncol=4, fontsize=8) + fig.tight_layout(rect=(0, 0.05, 1, 1)) + fig.savefig(FIG / "simulation_accuracy.png") + + ct = df[df.method == "tHMM-ctmc"] + fig, ax = plt.subplots(figsize=(4.2, 3)) + for i, sep in enumerate([1.0, 0.5]): + g = ct[ct.sep == sep].groupby("frac") + ax.errorbar( + g.Q_rel_err.mean().index, + g.Q_rel_err.mean(), + yerr=g.Q_rel_err.std(), + color=C[i], + lw=2, + marker="o", + ms=4, + capsize=2, + label=f"Q, {'well separated' if sep == 1 else 'overlapping'}", + ) + ax.set_xscale("log") + ax.set_xticks([0.1, 0.25, 0.5, 1.0], ["10%", "25%", "50%", "100%"]) + ax.set_xlabel("Leaves sampled") + ax.set_ylabel("‖Q̂ − Q‖ / ‖Q‖") + ax.legend() + fig.tight_layout() + fig.savefig(FIG / "simulation_Q_error.png") + + +def fig_scalability(): + df = pd.read_csv(RES / "scalability.csv") + fig, ax = plt.subplots(figsize=(4.2, 3)) + ax.loglog(df.n_nodes, df.estep_s, color=C[0], lw=2, marker="o", ms=4, label="one E-step") + ax.loglog(df.n_nodes, df.fit_s, color=C[1], lw=2, marker="o", ms=4, label="full EM fit") + x = np.array([df.n_nodes.min(), df.n_nodes.max()]) + ax.loglog(x, df.estep_s.iloc[-1] * x / x[-1], color=INK2, lw=0.8, ls="--", label="linear") + ax.set_xlabel("Tree nodes") + ax.set_ylabel("Seconds (one core)") + ax.legend() + fig.tight_layout() + fig.savefig(FIG / "scalability.png") + + +def fig_selection(): + df = pd.read_csv(RES / "model_selection.csv") + df["gain"] = df.heldout_ll - df.heldout_ll_no_tree + main = df[(df.bl_model == "mut") & (df.delta == 0.5)] + fig, axes = plt.subplots(1, 3, figsize=(10.5, 3.1)) + g = main.groupby("K") + axes[0].errorbar( + g.heldout_ll.mean().index, + g.heldout_ll.mean(), + yerr=g.heldout_ll.std(), + color=C[0], + lw=2, + marker="o", + ms=4, + capsize=2, + label="tHMM (tree + emissions)", + ) + axes[0].errorbar( + g.heldout_ll_no_tree.mean().index, + g.heldout_ll_no_tree.mean(), + yerr=g.heldout_ll_no_tree.std(), + color=C[1], + lw=2, + marker="o", + ms=4, + capsize=2, + label="same emissions, no tree", + ) + axes[0].set_title("Held-out log-likelihood (20% of leaves)", fontsize=9) + axes[0].legend(fontsize=7.5) + axes[1].errorbar( + g.gain.mean().index, g.gain.mean(), yerr=g.gain.std(), color=C[0], lw=2, marker="o", ms=4, capsize=2 + ) + axes[1].set_title("Lineage information gain (nats)", fontsize=9) + for ax in axes[:2]: + ax.set_xlabel("Hidden states K") + sens = df[df.K.isin([6, 8, 10])].copy() + sens["model"] = np.where(sens.bl_model == "unit", "unit", "n_mut + " + sens.delta.astype(str)) + order = ["n_mut + 0.1", "n_mut + 0.25", "n_mut + 0.5", "n_mut + 1.0", "n_mut + 2.0", "unit"] + for i, K in enumerate([6, 8, 10]): + s_ = sens[sens.K == K].groupby("model").heldout_ll.mean().reindex(order) + axes[2].plot(range(len(order)), s_.values - s_.max(), color=C[i], lw=2, marker="o", ms=4, label=f"K={K}") + axes[2].set_xticks(range(len(order)), order, rotation=30, ha="right") + axes[2].set_ylabel("held-out LL − best") + axes[2].set_title("Branch-length model", fontsize=9) + axes[2].legend(fontsize=7.5) + fig.tight_layout() + fig.savefig(FIG / "model_selection.png") + + +def cluster_names(): + with open(RES / "kptracer_inputs.pkl", "rb") as f: + recs = pickle.load(f) + lab = np.concatenate([r["leiden"] for r in recs]) + name = np.concatenate([r["cluster_name"] for r in recs]) + return {int(k): str(v) for k, v in zip(lab, name, strict=True)} + + +def fig_transitions(name="anchored"): + names = cluster_names() + with open(RES / f"fit_{name}.pkl", "rb") as f: + fit = pickle.load(f) + labels = [names[int(k)] for k in fit["state_labels"]] + thmm = pd.read_csv(RES / f"transitions_thmm_{name}.csv", index_col=0).values + fitch = pd.read_csv(RES / f"transitions_fitch_{name}.csv", index_col=0).values + Q = fit["Q"] + rates = Q.copy() + np.fill_diagonal(rates, np.nan) + fig, axes = plt.subplots(1, 3, figsize=(15, 5.2)) + for ax, M, title in [ + (axes[0], rates, "tHMM rates Q (log10)"), + (axes[1], thmm, "tHMM E[edge changes] (log10)"), + (axes[2], fitch, "Fitch changes (log10)"), + ]: + M = M.astype(float).copy() + np.fill_diagonal(M, np.nan) + im = ax.imshow(np.log10(M + (1e-3 if ax is axes[0] else 0.5)), cmap="Blues") + ax.set_xticks(range(len(labels)), labels, rotation=60, ha="right", fontsize=7) + ax.set_yticks(range(len(labels)), labels, fontsize=7) + ax.set_xlabel("to") + ax.set_ylabel("from") + ax.set_title(title, fontsize=9) + ax.grid(False) + fig.colorbar(im, ax=ax, shrink=0.7, label="log10") + fig.tight_layout() + fig.savefig(FIG / f"transitions_{name}.png") + + +def fig_benchmarks(name="anchored"): + tb = pd.read_csv(RES / f"tumor_benchmarks_{name}.csv") + from scipy.stats import spearmanr + + pairs = [ + ("fitch_plasticity", "Fitch plasticity (changes / edge)"), + ("published_mean_scPlasticity", "Published mean scPlasticity"), + ("path_autocorr_z_leiden", "PATH-style auto-correlation z (Leiden)"), + ] + fig, axes = plt.subplots(1, 3, figsize=(10, 3.2)) + for ax, (col, lab) in zip(axes, pairs, strict=True): + ax.scatter( + tb[col], tb.thmm_plasticity, s=np.clip(tb.n_cells / 20, 12, 120), color=C[0], edgecolor="white", lw=1 + ) + rho = spearmanr(tb[col], tb.thmm_plasticity)[0] + ax.set_xlabel(lab) + ax.set_ylabel("tHMM plasticity (E[changes] / edge)") + ax.set_title(f"Spearman ρ = {rho:.2f} (21 tumors)", fontsize=9) + fig.tight_layout() + fig.savefig(FIG / f"tumor_benchmarks_{name}.png") + + +def fig_unsup_contingency(): + names = cluster_names() + with open(RES / "fit_unsup.pkl", "rb") as f: + fit = pickle.load(f) + with open(RES / "kptracer_inputs.pkl", "rb") as f: + recs = pickle.load(f) + recs = [r for r in recs if r["tumor"] in fit["tumors"]] + lab = np.concatenate([r["leiden"] for r in recs]) + gam = np.vstack([g[r["tree"].is_leaf] for g, r in zip(fit["gammas"], recs, strict=True)]) + uniq = np.unique(lab) + M = np.array([gam[lab == c].sum(axis=0) for c in uniq]) + M = M / M.sum(axis=1, keepdims=True) + fig, ax = plt.subplots(figsize=(1.2 + 0.45 * M.shape[1], 5)) + im = ax.imshow(M, cmap="Blues", vmin=0, vmax=1, aspect="auto") + ax.set_yticks(range(len(uniq)), [names[int(c)] for c in uniq], fontsize=7) + ax.set_xticks(range(M.shape[1]), [f"S{k}" for k in range(M.shape[1])], fontsize=7) + ax.set_xlabel("tHMM hidden state") + ax.set_title("Posterior state mass per published cluster", fontsize=9) + ax.grid(False) + fig.colorbar(im, ax=ax, shrink=0.7) + fig.tight_layout() + fig.savefig(FIG / "unsup_vs_clusters.png") + + +if __name__ == "__main__": + import sys + + FIG.mkdir(exist_ok=True) + which = sys.argv[1:] or ["simulation", "scalability", "selection", "transitions", "benchmarks", "unsup"] + for w in which: + { + "simulation": fig_simulation, + "scalability": fig_scalability, + "selection": fig_selection, + "transitions": fig_transitions, + "benchmarks": fig_benchmarks, + "unsup": fig_unsup_contingency, + }[w]() diff --git a/analysis/kptracer/prepare_data.py b/analysis/kptracer/prepare_data.py new file mode 100644 index 000000000..fae456989 --- /dev/null +++ b/analysis/kptracer/prepare_data.py @@ -0,0 +1,120 @@ +"""Build per-tumor tree + embedding inputs from the KP-Tracer Zenodo release (record 5847462). + +Run with ``uv run --with anndata python analysis/kptracer/prepare_data.py [DATA_DIR]``; DATA_DIR defaults +to ``$KPTRACER_DATA`` or ``~/data/kptracer/KPTracer-Data``. Writes ``results/kptracer_inputs.pkl``. + +Filtering follows Yang et al. 2022 (Figure 4 notebook in KPTracer-release): sgNT primary tumors with a +tree, at least 100 cells, >5% unique alleles and >20% unsaturated targets; leaves restricted to cells in +the expression AnnData; clusters with <=2.5% of a tumor's cells dropped; unifurcations collapsed. +""" + +import os +import pickle +import sys +from pathlib import Path +from typing import Any + +import anndata as ad # ty: ignore[unresolved-import] (only needed here; run with `uv run --with anndata`) +import numpy as np +import pandas as pd + +from lineage.tree_io import ( + graph_to_phylotree, + lca_character_states, + mutation_branch_lengths, + newick_to_graph, + prune_graph, +) + +OUT = Path(__file__).parent / "results" +FILTER_PROP = 0.025 + + +def data_dir() -> Path: + if len(sys.argv) > 1: + return Path(sys.argv[1]) + return Path(os.environ.get("KPTRACER_DATA", Path.home() / "data/kptracer/KPTracer-Data")) + + +def load_character_matrix(path: Path) -> pd.DataFrame: + cm = pd.read_csv(path, sep="\t", index_col=0, dtype=str) + return cm.replace("-", "-1").astype(int) + + +def add_pca(adata, records, n_genes=2000, n_pcs=10): + """Sensitivity embedding: PCA of the log-normalized expression (not batch corrected), same dimension as scVI.""" + from sklearn.decomposition import PCA + + names = np.concatenate([r["leaf_names"] for r in records]) + rows = np.sort(adata.obs_names.get_indexer(names)) + X = adata.X[rows] + gene_var = np.asarray(X.power(2).mean(axis=0)).ravel() - np.asarray(X.mean(axis=0)).ravel() ** 2 + genes = np.argsort(gene_var)[::-1][:n_genes] + pcs = PCA(n_pcs, random_state=0).fit_transform(X[:, genes].toarray()) + lookup = pd.DataFrame(pcs, index=adata.obs_names[rows]) + for r in records: + r["pca"] = lookup.loc[r["leaf_names"]].values + + +def main(): + root = data_dir() + adata = ad.read_h5ad(root / "expression/adata_processed.nt.h5ad", backed="r") + obs = adata.obs[["Tumor", "leiden_sub", "Cluster-Name"]].copy() + scvi = pd.DataFrame(np.asarray(adata.obsm["X_scVI"]), index=adata.obs_names) + stats = pd.read_csv(root / "tumor_statistics.tsv", sep="\t", index_col=0) + published = pd.read_csv(root / "plasticity_scores.tsv", sep="\t", index_col=0) + + passing = stats[(stats.NumCells >= 100) & (stats.PercentUnique > 0.05) & (stats.PercentUnsaturatedTargets > 0.2)] + tumors = sorted( + t + for t in passing.index + if t.split("_")[1] == "NT" + and t.split("_")[2].startswith("T") + and len(t.split("_")) == 3 + and (root / f"trees/{t}_tree.nwk").exists() + ) + + records: list[dict[str, Any]] = [] + for tumor in tumors: + g = newick_to_graph(root / f"trees/{tumor}_tree.nwk") + leaves = [n for n in g if g.out_degree(n) == 0] + cells = obs.index.intersection(leaves) + cells = cells[obs.loc[cells, "Tumor"].astype(str).values == tumor] + counts = obs.loc[cells, "leiden_sub"].value_counts() + keep_clusters = counts.index[counts > FILTER_PROP * len(cells)] + cells = cells[obs.loc[cells, "leiden_sub"].isin(keep_clusters).values] + if len(cells) < 20: + continue + pt = graph_to_phylotree(prune_graph(g, cells, collapse_unifurcations=True)) + + cm = load_character_matrix(root / f"trees/{tumor}_character_matrix.txt") + chars = np.full((len(pt.names), cm.shape[1]), -1, dtype=int) + leaf_idx = np.nonzero(pt.is_leaf)[0] + chars[leaf_idx] = cm.loc[list(pt.names[leaf_idx])].values + anc = lca_character_states(pt, chars) + n_mut = mutation_branch_lengths(pt, anc) + + leaf_names = pt.leaf_names.astype(str) + records.append( + { + "tumor": tumor, + "tree": pt, + "n_mut": n_mut, + "leaf_names": leaf_names, + "scvi": scvi.loc[leaf_names].values, + "leiden": obs.loc[leaf_names, "leiden_sub"].astype(int).values, + "cluster_name": obs.loc[leaf_names, "Cluster-Name"].astype(str).values, + "published_scPlasticity": published.reindex(leaf_names)["scPlasticity"].values, + } + ) + print(tumor, len(leaf_names), len(pt.names), f"zero-mutation edges {np.mean(n_mut[1:] == 0):.2f}", flush=True) + + add_pca(adata, records) + OUT.mkdir(exist_ok=True) + with open(OUT / "kptracer_inputs.pkl", "wb") as f: + pickle.dump(records, f) + print(len(records), "tumors,", sum(len(r["leaf_names"]) for r in records), "cells") + + +if __name__ == "__main__": + main() diff --git a/analysis/kptracer/results/bdmm_thmm.json b/analysis/kptracer/results/bdmm_thmm.json new file mode 100644 index 000000000..adb0b6622 --- /dev/null +++ b/analysis/kptracer/results/bdmm_thmm.json @@ -0,0 +1,110 @@ +{ + "fit_seconds": 0.40406500396784395, + "bootstrap_seconds_200": 165.3083851109841, + "rates_true": [ + [ + -0.5249999999999999, + 0.44999999999999996, + 0.06, + 0.015 + ], + [ + 0.15000000000000002, + -0.4650000000000001, + 0.30000000000000004, + 0.015 + ], + [ + 0.015, + 0.06, + -0.31500000000000006, + 0.24 + ], + [ + 0.015, + 0.015, + 0.03, + -0.06 + ] + ], + "rates_mle": [ + [ + -0.6938094488252302, + 0.6926933042389103, + 0.0005692437833575629, + 0.0005469008029624074 + ], + [ + 0.27351277229404575, + -0.5356022788110938, + 0.2615896107371578, + 0.0004998957798901306 + ], + [ + 0.0004353445889864048, + 0.0010838638350814996, + -0.145226170955154, + 0.14370696253108609 + ], + [ + 0.002822984928211943, + 0.003868053560450177, + 0.011504328785367841, + -0.018195367274029962 + ] + ], + "rates_lo": [ + [ + -7.221626030149873, + 0.004712658739534418, + 0.00022665387602839124, + 0.00013246436381904553 + ], + [ + 0.0009838804107097083, + -6.567271320604512, + 0.001833156080395771, + 9.425506192124421e-05 + ], + [ + 9.065696115665095e-05, + 8.914236522078972e-05, + -3.7752944946946356, + 0.0003183062877254067 + ], + [ + 9.356910043729109e-05, + 0.0001043742745093608, + 9.884050344731869e-05, + -6.620462420705632 + ] + ], + "rates_hi": [ + [ + -0.15703392711992412, + 3.123378717138865, + 4.48993322499393, + 2.887389889511572 + ], + [ + 2.4763037979734364, + -0.1524985873490676, + 3.397957326556993, + 2.7520743068756914 + ], + [ + 0.2756951672250432, + 1.086556283740383, + -0.0010976466092711158, + 2.6930105727406772 + ], + [ + 2.7029590783413315, + 2.4406880475663124, + 4.97745474455694, + -0.0003985497187852798 + ] + ], + "coverage": 1.0, + "median_abs_log_ratio": 1.5127770836562071 +} \ No newline at end of file diff --git a/analysis/kptracer/results/benchmark_correlations.csv b/analysis/kptracer/results/benchmark_correlations.csv new file mode 100644 index 000000000..90d02fa55 --- /dev/null +++ b/analysis/kptracer/results/benchmark_correlations.csv @@ -0,0 +1,8 @@ +,thmm_plasticity~fitch_plasticity,thmm_plasticity~published_mean_scPlasticity,fitch_plasticity~published_mean_scPlasticity,thmm_plasticity~path_autocorr_leiden,fitch_plasticity~path_autocorr_leiden,thmm_plasticity~path_autocorr_thmm,thmm_plasticity~path_autocorr_z_leiden,fitch_plasticity~path_autocorr_z_leiden,cell_pooled_spearman,cell_median_within_tumor_spearman,transition_matrix_spearman_thmm_vs_fitch,total_changes_thmm,total_changes_fitch,total_jumps_ctmc,state_thmm_stay_prob~fitch_stay_prob,state_thmm_stay_prob~published_scPlasticity +anchored,0.9363636363636364,0.9350649350649349,0.9636363636363635,-0.15324675324675324,-0.05844155844155843,-0.22727272727272727,-0.14675324675324675,-0.011688311688311687,0.6492325148008402,0.450233985198588,0.7903327001025244,4124.583519267508,3888.0,7321.389162037991,0.8131868131868131,-0.47802197802197804 +unsup,0.5233766233766234,0.425974025974026,0.9636363636363635,0.04285714285714286,-0.05844155844155843,-0.0987012987012987,0.13376623376623376,-0.011688311688311687,-0.6650309605887625,0.29155208898688595,,,,,, +anchored_no3724,0.9308270676691727,0.9278195488721804,0.9578947368421054,-0.0015037593984962405,0.09022556390977443,-0.08721804511278194,-0.007518796992481202,0.13233082706766916,0.49588705158500235,0.4424230838794342,0.7360927454296489,4028.611114231635,3753.0,7304.706660438407,0.690909090909091,-0.13636363636363638 +anchored_delta0.1,0.9207792207792208,0.8818181818181818,0.9636363636363635,-0.23636363636363636,-0.05844155844155843,-0.3818181818181818,-0.15064935064935067,-0.011688311688311687,0.768982722610983,0.4722603692625214,0.776932549796172,4371.860948354631,3888.0,25549.6158847783,0.8516483516483516,-0.5164835164835164 +anchored_delta2,0.9506493506493506,0.9311688311688312,0.9636363636363635,-0.1064935064935065,-0.05844155844155843,-0.16753246753246753,-0.09480519480519481,-0.011688311688311687,0.4755375135778158,0.3403489212153558,0.7763527685873207,3717.0746371973637,3888.0,4732.1560877968,0.8461538461538461,-0.510989010989011 +anchored_unit,0.9519480519480519,0.9116883116883117,0.9636363636363635,-0.174025974025974,-0.05844155844155843,-0.21428571428571425,-0.15324675324675324,-0.011688311688311687,0.40223607797122185,0.4447046528818706,0.776786746829449,3517.74661220646,3888.0,4152.367170701685,0.8461538461538461,-0.510989010989011 +anchored_freeE,0.5155844155844156,0.4623376623376624,0.9636363636363635,0.11948051948051949,-0.05844155844155843,-0.03506493506493507,0.1012987012987013,-0.011688311688311687,0.5130955554979857,0.37579514930319224,0.6507186404846763,8228.489019158962,3888.0,11868.546728022078,0.4340659340659341,-0.3021978021978022 diff --git a/analysis/kptracer/results/cell_plasticity_anchored.csv.gz b/analysis/kptracer/results/cell_plasticity_anchored.csv.gz new file mode 100644 index 000000000..d33b2d3aa Binary files /dev/null and b/analysis/kptracer/results/cell_plasticity_anchored.csv.gz differ diff --git a/analysis/kptracer/results/cell_plasticity_unsup.csv.gz b/analysis/kptracer/results/cell_plasticity_unsup.csv.gz new file mode 100644 index 000000000..17fc23f5e Binary files /dev/null and b/analysis/kptracer/results/cell_plasticity_unsup.csv.gz differ diff --git a/analysis/kptracer/results/flows_anchored.csv b/analysis/kptracer/results/flows_anchored.csv new file mode 100644 index 000000000..6d88dd739 --- /dev/null +++ b/analysis/kptracer/results/flows_anchored.csv @@ -0,0 +1,14 @@ +from,to,jumps_fwd,jumps_rev,fwd_fraction,fwd_fraction_lo,fwd_fraction_hi,boot_same_direction +Gastric-like,High plasticity,1138.7669329749365,975.1180416812648,0.5387080880122834,0.41035414989305324,0.618032112279444,0.6041666666666666 +AT1-like,AT2-like,900.100774366233,835.3007361546552,0.5186700420100844,3.8372730420233035e-06,0.5651627187975976,0.6458333333333334 +Early gastric,AT1-like,686.4966791214887,38.186796527569705,0.9473055508912108,0.5462864021888804,0.9958804757165077,0.9791666666666666 +Early EMT-1,Pre-EMT,449.1511571151693,322.95824165569815,0.581719582523124,0.4090090546282165,0.6118266287470971,0.9166666666666666 +Late Gastric,Endoderm-like,293.4533305878584,164.5142146053141,0.640773202529185,0.5001135108211257,0.9999777056286336,0.9791666666666666 +AT2-like,High plasticity,145.7696625094948,53.86347856268116,0.7301876919163076,0.12973855048212768,0.9738640312167259,0.7708333333333334 +AT1-like,High plasticity,132.83254826364532,115.25647104387188,0.5354229245390081,0.0905724006968742,0.9974181302153879,0.7083333333333334 +Gastric-like,AT1-like,123.79808156204494,0.0062815288783193,0.9999492624595653,0.3525060444593861,0.9999938203891346,0.9166666666666666 +Early gastric,High plasticity,115.20677515419484,111.27571503681058,0.5086785078044421,0.23433421979213165,0.9999858895241799,0.7291666666666666 +Endoderm-like,High plasticity,100.82099732715206,32.26989453166403,0.7575349140653727,0.04161487677552819,0.9989825538956572,0.7916666666666666 +Early EMT-1,Early EMT-2,87.05711413499839,52.25469906866044,0.6249083414608227,0.0010034096683396068,0.6846797972935404,0.6458333333333334 +Mesenchymal-2,Mesenchymal-1,85.66658290448648,14.918351182231527,0.8516840387908405,0.49333806158979665,0.940767265497589,0.9375 +Gastric-like,Early EMT-1,57.1940302706902,0.6065580436458579,0.9895060230123054,0.0023376692346684506,0.9999763821416887,0.7083333333333334 diff --git a/analysis/kptracer/results/model_selection.csv b/analysis/kptracer/results/model_selection.csv new file mode 100644 index 000000000..5ce4c42b0 --- /dev/null +++ b/analysis/kptracer/results/model_selection.csv @@ -0,0 +1,85 @@ +K,rep,bl_model,delta,excluded,heldout_ll,heldout_ll_no_tree,train_ll,full_ll,BIC,n_params +2,0,mut,0.5,,-74294.64041801743,-78053.45445837022,-298239.69012129464,-372527.33847205795,745496.4135982655,43.0 +2,1,mut,0.5,,-74238.50132919778,-77998.22746738301,-298295.15869800013,,, +2,2,mut,0.5,,-75047.64661556786,-78825.20256414171,-297505.93070055,,, +3,0,mut,0.5,,-69401.13508232681,-74960.88161458499,-278594.7274074833,-347989.59150765877,696677.7433056006,68.0 +3,1,mut,0.5,,-69248.74339584899,-74802.03435087147,-278759.43324339605,,, +3,2,mut,0.5,,-70038.58043043711,-75659.00977754527,-277958.13070406107,,, +4,0,mut,0.5,,-66481.83677895217,-72458.04591715205,-266714.671621687,-333150.3811136664,667276.69204464,95.0 +4,1,mut,0.5,,-66238.68712331029,-72293.25859151753,-266954.0127039768,,, +4,2,mut,0.5,,-67056.2359669068,-73209.71649794622,-266132.38029548834,,, +5,0,mut,0.5,,-65234.00152455931,-71787.5209913768,-261990.5289485321,-327201.0628982676,655675.9710317571,124.0 +5,1,mut,0.5,,-65007.29461124602,-71627.9138050791,-262217.193030657,,, +5,2,mut,0.5,,-65158.0525937456,-72172.06115343323,-258828.2795298286,,, +6,0,mut,0.5,,-64470.359934557186,-71185.08427798685,-259019.18545835002,-323460.3308093877,648512.9681628029,155.0 +6,1,mut,0.5,,-64277.88909870729,-71090.55639559457,-259207.03876590705,,, +6,2,mut,0.5,,-64465.702530543625,-71859.99170284692,-255814.73536420232,,, +7,0,mut,0.5,,-62902.259538623155,-70361.90870919543,-252358.16082820046,-314236.7563248607,630404.8263934449,188.0 +7,1,mut,0.5,,-62718.402846683275,-70198.62014557893,-252564.61670369076,,, +7,2,mut,0.5,,-63387.30874447168,-70871.68659554972,-252261.4663639514,,, +8,0,mut,0.5,,-62147.89826822956,-69711.69274800894,-249786.0344027036,-311895.64623095596,626082.1592962224,223.0 +8,1,mut,0.5,,-62025.14643106221,-69557.98633753543,-250564.42098874203,,, +8,2,mut,0.5,,-62622.48631220537,-70291.07287916186,-249309.4157239883,,, +9,0,mut,0.5,,-61558.158686410985,-68973.61927575174,-247278.52665016157,-309057.62431262445,620786.2144410368,260.0 +9,1,mut,0.5,,-60656.815530352826,-68849.41005129753,-244646.8008084108,,, +9,2,mut,0.5,,-61531.10743944729,-69879.02438697872,-244267.40183125806,,, +10,0,mut,0.5,,-60952.37255083744,-68726.96204845428,-244721.42154560526,-300436.14697117824,603943.9046305126,299.0 +10,1,mut,0.5,,-60429.00635981239,-68625.8231912647,-242957.5461924876,,, +10,2,mut,0.5,,-60510.46014947717,-69201.15838476118,-240021.1644899825,,, +11,0,mut,0.5,,-59499.63112460147,-68081.70358415182,-239204.5562780734,-300715.90707260865,604924.6155966324,340.0 +11,1,mut,0.5,,-59182.065509119035,-67892.9749883969,-238717.73837317553,,, +11,2,mut,0.5,,-60436.95280546007,-68793.59448602432,-240085.65961801878,,, +12,0,mut,0.5,,-58628.39078093326,-67618.97375886112,-235620.94164347672,-295438.04778964684,594810.6336848582,383.0 +12,1,mut,0.5,,-59284.71042203491,-67661.24566464536,-238687.83453115306,,, +12,2,mut,0.5,,-59260.49806992321,-68343.11349383071,-235117.26352381756,,, +13,0,mut,0.5,,-58127.74545691978,-67131.00127039535,-233498.3231783072,-293861.7182120844,592120.2570747735,428.0 +13,1,mut,0.5,,-58331.37384161451,-67016.20222239582,-234981.72287346303,,, +13,2,mut,0.5,,-58975.559467201885,-67834.66799755198,-234013.31919229127,,, +14,0,mut,0.5,,-58104.75727675505,-66821.49818487234,-233346.1657766598,-288964.9040106532,582809.4571078421,475.0 +14,1,mut,0.5,,-57292.565558799375,-66563.10842346345,-231552.80310333136,,, +14,2,mut,0.5,,-58229.54770741601,-67356.74430698463,-230850.22270737894,,, +6,0,mut,0.1,,-64559.97061145412,-71181.5486814052,-259342.14386019635,-323871.7890631589,649335.8846703452,155.0 +6,1,mut,0.1,,-64373.637438223486,-71092.30748325353,-259564.10575459024,,, +6,2,mut,0.1,,-64642.71799454739,-71855.62943186388,-256328.34778755877,,, +6,0,mut,0.25,,-64520.65107590111,-71184.09835874804,-259186.78747947048,-323678.7859545871,648949.8784532016,155.0 +6,1,mut,0.25,,-64325.39091600629,-71092.32060193257,-259385.3662759467,,, +6,2,mut,0.25,,-64520.508155943535,-71860.62499668924,-256000.27823257592,,, +6,0,mut,1.0,,-64619.833962639685,-71300.3649146285,-259600.12025404724,-323281.5753860831,648155.4573161936,155.0 +6,1,mut,1.0,,-64240.83933923789,-71089.8031464663,-259069.79874351335,,, +6,2,mut,1.0,,-64424.09093530961,-71858.6854165317,-255664.98678657782,,, +6,0,mut,2.0,,-64582.929564194565,-70934.67496058915,-259587.27569021343,-323145.537896648,647883.3823373234,155.0 +6,1,mut,2.0,,-64213.921601784445,-71088.99343635613,-258970.3437103005,,, +6,2,mut,2.0,,-64385.86266706569,-71859.02709644714,-255533.91932359527,,, +6,0,unit,0.0,,-64494.92276029521,-70933.97692856085,-259269.67529339928,-322818.26363231224,647228.8338086519,155.0 +6,1,unit,0.0,,-64156.901365067744,-71093.85954262073,-258702.75696995275,,, +6,2,unit,0.0,,-64294.02679203705,-71860.33516160584,-255286.23158523784,,, +8,0,mut,0.1,,-62142.258076977545,-69679.92008489836,-249821.00807589636,-313171.7070268511,628634.2808880126,223.0 +8,1,mut,0.1,,-62113.0243552276,-69900.31038344964,-250613.03932462196,,, +8,2,mut,0.1,,-62746.543372033695,-70354.67244333352,-249663.7074404146,,, +8,0,mut,0.25,,-62138.28880603935,-69578.93482931299,-249740.72478657824,-312077.26848938764,626445.4038130858,223.0 +8,1,mut,0.25,,-62103.556846003834,-69584.98695394669,-250772.21578994824,,, +8,2,mut,0.25,,-62679.89523000635,-70322.33242338499,-249465.98763360863,,, +8,0,mut,1.0,,-62103.49462831415,-69685.21629639984,-249658.55321365397,-311735.39487549744,625761.6565853053,223.0 +8,1,mut,1.0,,-61945.748491349805,-69533.83727404948,-250415.07235871776,,, +8,2,mut,1.0,,-62577.409572877,-70270.78685749855,-249184.96811526056,,, +8,0,mut,2.0,,-62087.64547835886,-69673.18488306756,-249572.20204284383,-311622.9905791752,625536.8479926608,223.0 +8,1,mut,2.0,,-61912.078300312765,-69526.64916110394,-250315.50577807857,,, +8,2,mut,2.0,,-62550.95084834859,-70262.3224654827,-249103.9981139754,,, +8,0,unit,0.0,,-62022.450285465304,-69634.08104466276,-249469.28414468406,-311443.08034648496,625177.0275272804,223.0 +8,1,unit,0.0,,-61844.41485643921,-69553.0234024545,-249680.39833843615,,, +8,2,unit,0.0,,-62535.67580367729,-70262.13041857837,-248942.35954833459,,, +10,0,mut,0.1,,-61021.1472429728,-68723.92204550398,-245017.3273254776,-302098.61122414516,607268.8331364464,299.0 +10,1,mut,0.1,,-60522.12875840773,-68586.88076170962,-243443.50837885492,,, +10,2,mut,0.1,,-60810.750333578326,-69332.58106143743,-241338.10049339975,,, +10,0,mut,0.25,,-60985.7229149524,-68723.77566906101,-244871.1922214941,-300759.9808878088,604591.5724637737,299.0 +10,1,mut,0.25,,-60472.852110927706,-68614.99417817473,-243163.98437633144,,, +10,2,mut,0.25,,-60726.5623038888,-69323.06677468901,-241060.55993015613,,, +10,0,mut,1.0,,-60913.49588717986,-68729.99510411348,-244580.9176952854,-300217.4801030564,603506.5708942689,299.0 +10,1,mut,1.0,,-60401.12227065018,-68627.67693882737,-242812.4569689422,,, +10,2,mut,1.0,,-60471.787319278454,-69195.30888938719,-239839.50673971415,,, +10,0,mut,2.0,,-60909.244801619425,-68731.27222789712,-244486.43622211006,-300098.2094723102,603268.0296327765,299.0 +10,1,mut,2.0,,-60380.98656473147,-68625.36842811674,-242727.4693461849,,, +10,2,mut,2.0,,-60451.90930207925,-69233.46121168323,-239691.349690477,,, +10,0,unit,0.0,,-60899.637084662325,-68733.39915193948,-244404.28810193873,-300050.48594102776,603172.5825702116,299.0 +10,1,unit,0.0,,-60358.81749826607,-68626.72822747489,-242677.70597663432,,, +10,2,unit,0.0,,-60446.65275324182,-69233.59764988684,-239646.9700083738,,, diff --git a/analysis/kptracer/results/scalability.csv b/analysis/kptracer/results/scalability.csv new file mode 100644 index 000000000..c798aaa6b --- /dev/null +++ b/analysis/kptracer/results/scalability.csv @@ -0,0 +1,7 @@ +n_leaves,n_nodes,estep_s,mstep_s,fit_s,iters +100,199,0.003971465339418501,0.0012247519916854799,0.27184900897555053,52 +300,599,0.011508099676575512,0.003191747993696481,0.5940438139950857,41 +1000,1999,0.03742653568042442,0.008937962003983557,1.113913269015029,25 +3000,5999,0.11597497334393363,0.03329339803894982,5.726646336959675,37 +10000,19999,0.38040964499426383,0.07909284095512703,9.572494189953431,19 +30000,59999,1.1792791943492678,0.2758738160482608,33.904486912011635,24 diff --git a/analysis/kptracer/results/simulation.csv b/analysis/kptracer/results/simulation.csv new file mode 100644 index 000000000..673bc13de --- /dev/null +++ b/analysis/kptracer/results/simulation.csv @@ -0,0 +1,161 @@ +method,sep,frac,seed,n_leaves,leaf_acc,anc_acc,mean_rmse,changes_ratio,plasticity_corr,iters,Q_rel_err,exit_rate_rel_err,big_rate_rel_err,small_rate_abs_err,pi_tv,q01,q02,q03,q10,q12,q13,q20,q21,q23,q30,q31,q32 +tHMM-ctmc,1.0,1.0,0,10000,0.9795999999999999,0.9808617234468937,0.027270961615375046,0.9951087544293691,0.9649621495934767,22.0,0.07664727059797574,0.18733855779712,0.060664940039500764,0.0043874694274854,0.11266081722436815,0.1445375521547187,0.018782793528942833,0.0045559645547484495,0.044327844999028754,0.09704141138164331,0.003125852708287384,5.8623646717042736e-05,0.026413912537048796,0.07494284360108247,0.014102302944692674,0.012069582742651104,0.005962808365813665 +tHMM-discrete,1.0,1.0,0,10000,0.9763000000000002,0.9758517034068136,0.027026620619816,0.983395909690599,0.966151095076841,15.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,1.0,0,10000,0.8658999999999999,0.9463927855711425,,1.7724867724867726,0.34673830368765934,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,1.0,0,10000,1.0,0.9760521042084168,,0.9327286470143613,0.973506532583945,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,1.0,1,10000,0.9732,0.9825651302605211,0.020917196061156128,1.0136257423945678,0.9888896574640077,15.0,0.09175905034607441,0.07458725438132263,0.07071651967401751,0.0021091892474675196,0.021159374748742257,0.14977827046311387,0.024089620487236445,0.004619541235603371,0.051822942240053224,0.0929762734519114,0.0037692712431265803,0.0034705371570587645,0.012440206177647867,0.06602465827987636,0.006214368244598624,0.004268135321751953,0.009862783616906377 +tHMM-discrete,1.0,1.0,1,10000,0.968,0.9756513026052105,0.021702621665402006,1.007230669718419,0.9874659015052503,11.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,1.0,1,10000,0.9197000000000001,0.9559118236472945,,1.4741100323624596,0.9393404814288803,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,1.0,1,10000,1.0,0.975250501002004,,0.9457928802588996,0.9907798194301086,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,1.0,2,10000,0.9700999999999999,0.9746492985971942,0.029165619315235126,1.0031711068201994,0.9613172238632921,20.0,0.07808369261553864,0.04805719617891546,0.0676785000205031,0.004445661397534076,0.05621789700125286,0.15106628308700387,0.02531333825435199,0.004434166919527554,0.054956848945367266,0.09641573514189855,0.002732653935400046,0.0038600319505459513,0.026621538949929174,0.09029006549442346,0.0023035212888797025,0.014858917577617824,0.0028981295072731234 +tHMM-discrete,1.0,1.0,2,10000,0.9669000000000001,0.9705410821643286,0.029132015397396124,0.9898800975752179,0.9396708969537713,13.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,1.0,2,10000,0.8738999999999999,0.939879759519038,,1.6697841726618705,0.6920872516827042,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,1.0,2,10000,1.0,0.9705410821643286,,0.923021582733813,0.9450960862895056,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,1.0,3,10000,0.9894000000000001,0.9821643286573145,0.02592810285810201,0.997763722839509,0.9497050405356138,15.0,0.10426316080934821,0.1893960278176779,0.08062616804705791,0.005018998439901839,0.07769936155394797,0.14032893914220876,0.02399062351198652,0.003379621115059897,0.05524265477782615,0.1093212414264626,0.006353214405121581,0.007550453332694883,0.009687921933131949,0.07520276613478871,0.0048258068711112346,0.020098279139341813,0.004947232950627005 +tHMM-discrete,1.0,1.0,3,10000,0.9884999999999999,0.981062124248497,0.026417533335916316,0.9909865478815862,0.9399023819613139,11.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,1.0,3,10000,0.9675999999999998,0.968937875751503,,1.1271676300578035,0.8371307145901804,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,1.0,3,10000,1.0,0.9745490981963927,,0.9364161849710982,0.9779849809263707,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,1.0,4,10000,0.9911,0.9837675350701403,0.032519788311975655,1.0102943775495379,0.9850283748267799,14.0,0.08339313708225081,0.2364339048698439,0.08488385560596762,0.002878172312536296,0.035233164995664104,0.15646939067390578,0.02065522826500509,0.0035535913090348663,0.05853822870443381,0.10642230111097915,0.001793498347548227,0.0038105921043592524,0.020602891211336228,0.08491348528528249,0.001990867886804081,0.0020521230327712903,3.206829553323055e-05 +tHMM-discrete,1.0,1.0,4,10000,0.99,0.9777555110220438,0.03230732917328664,1.0003144387851868,0.9776767790266596,10.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,1.0,4,10000,0.968,0.9711422845691382,,1.124113475177305,0.9377732018131583,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,1.0,4,10000,1.0,0.9750501002004007,,0.9276595744680851,0.9839615075351457,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.5,0,4912,0.975677588798718,0.9685534955091244,0.03335884075998768,0.9983432809492507,0.9436742625127045,23.0,0.08787380900260497,0.0649199845806541,0.07648218634794762,0.0034218748190854194,0.07185326846242504,0.1467067652465252,0.019164978132936574,0.01282932658108954,0.04997794078124313,0.08667462286004116,0.010210718121982246,0.005996093359790599,0.014175419388295904,0.06797768869915471,0.006199927995373704,0.001915893144602108,0.012395223160281852 +tHMM-discrete,1.0,0.5,0,4912,0.971443301494886,0.9525585589318902,0.033300201711863486,0.9843714863561132,0.918328694038689,13.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.5,0,4912,0.8947073842059444,0.9412616755342429,,1.3677639046538026,0.7008758627670453,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.5,0,4912,1.0,0.9550846756392397,,0.9171396140749148,0.9556335519894886,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.5,1,4991,0.9654354358641738,0.9735091257955648,0.037606531301540215,0.9867571894274247,0.9555257615291118,19.0,0.11531843624308459,0.20538650507337672,0.13340535684698715,0.0025046988038285875,0.22781617565543036,0.13615644324381165,0.02492547001524474,0.0063455658566447725,0.04306520428119074,0.09226931895555962,0.00638748574033608,0.004999532675414177,0.022421036462455045,0.09802626593542166,0.0011557584094588696,0.0016051414058628896,0.007281535153316002 +tHMM-discrete,1.0,0.5,1,4991,0.9606266176977185,0.966088832536224,0.037274269217513575,0.9701924183411467,0.9486510212915982,10.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.5,1,4991,0.9149988697369291,0.9436319171935684,,1.2813953488372094,0.9144223425361641,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.5,1,4991,1.0,0.9647975917120419,,0.9174418604651163,0.974712884607713,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.5,2,4969,0.9601171710208393,0.9639713613161612,0.03938620142249347,1.0219930098549013,0.9697104757619378,24.0,0.06267414457125031,0.035077120209262125,0.08850894864302661,0.002968203678264756,0.017569479154584917,0.1556483794054681,0.022232153847335547,0.005729635958637632,0.0589307672417527,0.0901201864649483,0.0056454512524564465,0.005848466404181128,0.022507267239855716,0.07688268386532683,3.100475136522956e-05,0.002446449819662489,0.0192601092946793 +tHMM-discrete,1.0,0.5,2,4969,0.9542090566852055,0.9441857008204749,0.04097611786909688,0.9968129801692049,0.9839870572184872,15.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.5,2,4969,0.8710096085209681,0.9149295878244568,,1.4369034994697774,0.6815813999619695,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.5,2,4969,1.0,0.9455655693570788,,0.911983032873807,0.972962713884736,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.5,3,5083,0.9860452131360706,0.9714082864858445,0.035561365283663576,1.0034467796480397,0.9885590885698672,20.0,0.06130694583650004,0.13062698991030922,0.042759623388675616,0.003612295160354906,0.049469368296276,0.15101602681049334,0.019605881258505195,0.002212759455416999,0.050375543350983494,0.1031585723794379,0.0035614686013612715,0.008358283314581431,0.008630689453456618,0.09001347125365586,0.0018741781548797622,0.004146385404591934,0.004428559703530402 +tHMM-discrete,1.0,0.5,3,5083,0.9864153037693777,0.9715842328335172,0.03610261349423583,0.9798742055107066,0.9853050297238481,13.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.5,3,5083,0.9706316076254208,0.9499090378616917,,1.0306681270536693,0.9779775448777682,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.5,3,5083,1.0,0.9531841533285247,,0.9090909090909091,0.9879774966661098,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.5,4,4969,0.9901882663756825,0.9789704722820296,0.044852173859343154,1.00185217939587,0.9154618141147133,16.0,0.08012156690023516,0.10499565320412252,0.09179717055055472,0.0028804212680575286,0.09474457459576471,0.14056484424901367,0.023730979401085,0.006161097101133356,0.044577675754989984,0.09133076409448777,0.002107017540217427,0.0023435017993632566,0.014693799160891198,0.088731903992559,0.005876619028846975,5.240821608949819e-05,0.008528598670043725 +tHMM-discrete,1.0,0.5,4,4969,0.9901639084986009,0.9696493896545437,0.04464259580505772,0.9933796662784258,0.9252649172294518,16.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.5,4,4969,0.965670502839725,0.9433863077206853,,1.0752565564424172,0.830631762579888,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.5,4,4969,1.0,0.9542226560235768,,0.9144811858608894,0.9202555539100199,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.25,0,2470,0.9648653143110298,0.9449155978903558,0.0592284915485013,1.0050626219645176,0.9441779264728676,44.0,0.15308932059734273,0.337541127738202,0.15466723489933523,0.004274232445704235,0.02902104387335138,0.16345046749959938,0.020969208723551184,0.008747006381364105,0.0669121358435842,0.10808928775524822,0.008220363042230318,4.644446200255436e-05,0.016744018407194427,0.07121091505259902,0.00014343127763941234,0.0016426187956191103,0.00016620563905621924 +tHMM-discrete,1.0,0.25,0,2470,0.9623561800823799,0.9299310901041034,0.061145030146043264,1.0045469317956943,0.9371532577612011,34.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.25,0,2470,0.8876278291898749,0.8848494949128135,,1.153731343283582,0.7675466720099954,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.25,0,2470,1.0,0.9074987580341297,,0.8835820895522388,0.9316362828549893,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.25,1,2399,0.9562677125820926,0.9383815925976859,0.051039025844328956,1.0091634864407724,0.8315008050857641,27.0,0.13492988386093754,0.3011324501470145,0.1626096667632099,0.004625468588092083,0.03987386390899311,0.15183737838657202,0.013896371122680004,0.004659231486543071,0.04249761114575574,0.11624746366072257,0.0022242314534523452,0.002604541215321009,0.014035541889562136,0.10605336512935322,3.404929082757294e-05,2.7623340031058343e-05,0.000514661496846147 +tHMM-discrete,1.0,0.25,1,2399,0.9477873944609033,0.9260280585484086,0.04946355247654032,0.9747200541120234,0.7719511533505835,19.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.25,1,2399,0.9103861411295064,0.8766159295585367,,1.137391304347826,0.755835598373703,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.25,1,2399,1.0,0.9056038302558556,,0.8626086956521739,0.857094942877925,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.25,2,2525,0.9514991900521628,0.9601994708458721,0.055669193195457184,0.9893541470995165,0.8624211015498797,24.0,0.19098465062321088,0.5699548352222474,0.09229791372508811,0.007552356257616967,0.07433226927095557,0.12563259476806352,0.014436466231074524,0.008417976084953673,0.04507612019629479,0.10143162296057281,0.00722856613200906,0.004949330541355923,0.009875437393454862,0.07248412311939118,0.014892509034063264,0.013769673505518486,0.030371359470276558 +tHMM-discrete,1.0,0.25,2,2525,0.948552731609477,0.9502003804029149,0.056232650499430076,0.9608306247263197,0.8532056951865078,15.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.25,2,2525,0.8791761407256315,0.9225211200813301,,1.2877959927140255,0.7063862989564562,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.25,2,2525,1.0,0.9434859203684265,,0.9052823315118397,0.9263711301759349,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.25,3,2538,0.9849912082849508,0.9461215326562133,0.05269098521589603,0.9675424693654355,0.9568294436051736,19.0,0.09853561423451791,0.15984139263836825,0.08134080467798135,0.0038777181864165793,0.11206123025301858,0.13354793976025725,0.020592102289656508,0.00302338971939999,0.05300180313585472,0.08986207033319588,0.005413601141124141,0.0009721820844282136,0.023590613862415358,0.07565860338171962,1.7235138789864225e-05,0.010724892907096339,0.00028665776634164054 +tHMM-discrete,1.0,0.25,3,2538,0.9862480588933143,0.9336046347626537,0.052426001225885276,0.9651261141556786,0.9582026482461034,15.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.25,3,2538,0.9710427838596939,0.9145929220764708,,0.9486301369863014,0.9236895362063855,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.25,3,2538,1.0,0.9160711951544996,,0.8681506849315068,0.9304184467394481,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.25,4,2534,0.9855530458130559,0.9675704859439433,0.052797939767783804,1.001063395460158,0.8918992457238021,20.0,0.1191983116985778,0.12540043474412918,0.12149781217250463,0.005939878678585576,0.10403828300631808,0.14022581189235517,0.015192639929295293,0.004994345615463983,0.05424807112899225,0.10864023031902134,0.004838437919841382,3.2479595890451436e-05,0.034251503366491125,0.06004269849048042,0.009936345661381782,0.014071245370045485,0.0006821619087426786 +tHMM-discrete,1.0,0.25,4,2534,0.9842926433366695,0.9461446381023901,0.05243679337084702,0.9878865163372458,0.8023910352567253,14.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.25,4,2534,0.9695053886603132,0.9363275663735934,,0.9883720930232558,0.7894341394047902,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.25,4,2534,1.0,0.9430307499621534,,0.8953488372093024,0.7965441209561843,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.1,0,1058,0.9537928404011545,0.9355098498568376,0.06911798396516794,1.0131342181373424,0.943795170227788,50.0,0.23297490620185532,0.7231779327979548,0.11376267425969488,0.011640696610134275,0.0707878010582845,0.15787786552119587,0.020757699399707976,9.499904510805199e-06,0.054720729298590474,0.11022891723409645,0.004008101904617394,0.0005467211762089832,0.0021089973204912473,0.0964662268201093,0.02880239787079386,0.00023791636881240067,0.045476712285213196 +tHMM-discrete,1.0,0.1,0,1058,0.9561488772335925,0.8595729023944919,0.06824457689138974,1.019248296480808,0.8845579417433679,54.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.1,0,1058,0.9198351071994836,0.8592679489371147,,0.9970760233918129,0.6018268953218584,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.1,0,1058,1.0,0.8868836042870093,,0.8450292397660819,0.879535142095998,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.1,1,957,0.9394498333797114,0.9113903288352316,0.07881894641474563,0.9210383380636152,0.8699278951341071,22.0,0.20032877513281208,0.169111124864666,0.23687797865548937,0.010707803677205273,0.06586588476702546,0.12711423750449577,0.0308226103102622,0.0101598345482809,0.02396479153133552,0.0887773624458684,0.011236485587063093,0.00012070688286812037,0.052915652480306864,0.06703923042101412,3.610231973247103e-05,7.244214574992117e-05,0.02575709784007963 +tHMM-discrete,1.0,0.1,1,957,0.9401555092792571,0.8623414792256346,0.07801499354381448,0.9263454509734518,0.89306287213007,20.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.1,1,957,0.895804363329505,0.8286113737751236,,0.9596273291925466,0.8430451917143524,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.1,1,957,1.0,0.8638403458239476,,0.8167701863354038,0.8968497560813873,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.1,2,1029,0.945995081581246,0.939392279998914,0.07119499513643511,0.9709628199318268,0.8954135583072378,49.0,0.16366919576252498,0.27148017902065713,0.17013695198793422,0.006497130646104986,0.10843413476978973,0.1380988887475357,0.021842088316048745,0.008954728010930509,0.031340065648555765,0.10745820456604713,0.0037714084233665215,0.0014408467226302427,0.0012847512821752163,0.06772589331365714,0.020765724292874963,7.948229587350436e-05,0.011990993273031152 +tHMM-discrete,1.0,0.1,2,1029,0.946284383925169,0.7734143971626353,0.0718651530485137,1.0723059981084784,0.8636991152332679,19.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.1,2,1029,0.8930916347643736,0.8558975924748682,,1.05625,0.7524888764126606,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.1,2,1029,1.0,0.8892451633640748,,0.8625,0.9493886788861248,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.1,3,1079,0.9902350384598284,0.9282329802778893,0.05363699207369255,0.9435091157344001,0.9470907141494267,25.0,0.09226823771084619,0.0702712223117248,0.13252631918715374,0.004248142175453865,0.24432548366988802,0.1517211962670199,0.03160975767987165,0.00235901787849225,0.06107263749601054,0.08984488835210427,0.0035263610836619385,0.005466612781693811,0.013894981146513458,0.09565014148554514,0.009633947039076217,0.0023731844266969174,0.005571635561646197 +tHMM-discrete,1.0,0.1,3,1079,0.990264204209333,0.9015054405846268,0.05404072221228289,0.9665380584893636,0.9353627227941465,15.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.1,3,1079,0.9744513589548129,0.8806743604091961,,0.8628048780487805,0.9493399883466921,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.1,3,1079,1.0,0.8870813418855606,,0.823170731707317,0.9441973572150597,,,,,,,,,,,,,,,,,, +tHMM-ctmc,1.0,0.1,4,993,0.9801931532391721,0.936473805207544,0.058691952257305884,0.9606290415541223,0.9196790343854749,20.0,0.19336925190843254,0.32833033407971535,0.14306758549332682,0.006237319241671412,0.16905536749762348,0.11520782748925862,0.01468851558341919,0.009388444702885395,0.0528149353675926,0.07745735570297203,0.00841382527892705,5.163394233034101e-05,0.032255728767950624,0.07531221000696801,3.4683680640771914e-05,5.581893727616943e-05,0.0003287926727252938 +tHMM-discrete,1.0,0.1,4,993,0.9782848881359634,0.8934329401867211,0.059270504498144964,0.9321245267494017,0.8680832402793278,24.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,1.0,0.1,4,993,0.9728258948201705,0.8665511580123537,,0.8692579505300353,0.8457447885854827,,,,,,,,,,,,,,,,,, +parsimony-oracle,1.0,0.1,4,993,1.0,0.8737512495874451,,0.8303886925795053,0.9060833916085875,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,1.0,0,10000,0.8936999999999999,0.9387775551102203,0.027064163639275718,0.984556648635528,0.8890553109630852,75.0,0.08845239743108964,0.13018191500820928,0.07375170885712426,0.005030651595750888,0.1149671595276371,0.1461313521496057,0.01865868179946631,0.0038470258013578716,0.04885424039638211,0.09182050229990635,0.002133557395085031,1.5520540321571763e-05,0.03364639967657847,0.06683954554512742,0.0021825201046800705,0.009498112414806442,0.0010619936844669458 +tHMM-discrete,0.5,1.0,0,10000,0.8875,0.927054108216433,0.030215927802188482,0.9940165690407753,0.8514626664059387,115.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,1.0,0,10000,0.6253000000000001,0.8267535070140279,,3.0634920634920637,0.34374286318753006,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,1.0,0,10000,1.0,0.9760521042084168,,0.9327286470143613,0.973506532583945,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,1.0,1,10000,0.8866000000000002,0.9311623246492987,0.02233050543508476,1.0271765147663245,0.9722921068600556,25.0,0.1179597928000813,0.16871004969642503,0.14538743619790287,0.0013155680672869395,0.025513884883525222,0.14956429514294525,0.020524258151215603,0.004630730849450385,0.062106928457297046,0.09703366056374782,0.0043630593193892085,0.005054511641324086,0.020324189030421085,0.05545255342111063,0.005425734658375838,0.001171712085518838,0.005638646688682663 +tHMM-discrete,0.5,1.0,1,10000,0.8747,0.9210420841683368,0.025311563834748092,1.0296690262146586,0.9695262696718427,22.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,1.0,1,10000,0.6373,0.8241482965931863,,3.1723300970873787,0.9046516799645422,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,1.0,1,10000,1.0,0.975250501002004,,0.9457928802588996,0.9907798194301086,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,1.0,2,10000,0.8723000000000001,0.9234468937875752,0.033669899627512816,1.0083546747281678,0.9037750927003055,43.0,0.09154946656664693,0.07838508325243554,0.057925570177311925,0.005560670404110913,0.06475169001362513,0.15813033326824544,0.02403274843314019,0.001782303439227092,0.04530818245595857,0.09782413540325144,0.006620213031941099,0.004183320635035312,0.03336315516671656,0.08495240496581044,0.00032625535374586226,0.015098303339976832,0.0033371773108791126 +tHMM-discrete,0.5,1.0,2,10000,0.8588999999999999,0.9111222444889778,0.035300452173031745,1.0044999354359492,0.884684927529941,34.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,1.0,2,10000,0.5809999999999998,0.7381763527054108,,3.0712230215827336,0.747009365393899,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,1.0,2,10000,1.0,0.9705410821643286,,0.923021582733813,0.9450960862895056,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,1.0,3,10000,0.9321999999999999,0.9589178356713427,0.028662993555660692,0.9894244507937737,0.8403982982272725,33.0,0.08145234324315005,0.08106482287514633,0.057765116503105424,0.004406016595828368,0.0771365172353825,0.13769855335789483,0.025775298272659806,0.0015582395369238401,0.05115824816966505,0.10959176582731806,0.005608926695420378,0.002033939257998269,0.023263154166263844,0.07760254399478086,0.000826356802750167,0.01633659492327798,0.006317305693322784 +tHMM-discrete,0.5,1.0,3,10000,0.9201,0.9503006012024049,0.02929356239301035,0.9773411328577488,0.8148593056145483,19.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,1.0,3,10000,0.7311,0.873246492985972,,2.4544797687861273,0.24119712821891742,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,1.0,3,10000,1.0,0.9745490981963927,,0.9364161849710982,0.9779849809263707,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,1.0,4,10000,0.9232000000000001,0.9531062124248496,0.03528458330948652,1.0043764288730366,0.9544330290951021,32.0,0.08967767578673493,0.2541171790033333,0.08488817341389787,0.003945646360583586,0.027589574581743864,0.15934940026673453,0.016818138925993275,0.0077012797994308844,0.056981343430873174,0.09984615515143061,0.0010087482864075666,0.002108586954021689,0.018847471587460956,0.09088464331526966,8.674846837687634e-05,0.0021516189150033982,0.00011479577749843594 +tHMM-discrete,0.5,1.0,4,10000,0.9137000000000001,0.9439879759519038,0.03476358373182147,1.0027113003886428,0.9461255620331438,22.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,1.0,4,10000,0.7051000000000001,0.8706412825651302,,2.5829787234042554,0.7635599486895974,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,1.0,4,10000,1.0,0.9750501002004007,,0.9276595744680851,0.9839615075351457,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.5,0,4912,0.8690927886676064,0.9263756659568712,0.04048739081510899,0.9830870826639875,0.710812573488758,58.0,0.16919490707944512,0.22280029889535644,0.14900260016388528,0.008478755206421284,0.0710726149293839,0.14312605311295248,0.012094450648597594,0.0217104545283304,0.04366127262114196,0.07423930888655592,0.013792763437759169,0.018373699898873237,0.021978153341434437,0.09326421034908981,0.01070417613141409,3.79663810458391e-05,0.018403211343202372 +tHMM-discrete,0.5,0.5,0,4912,0.8596466502746498,0.8967914252295147,0.04216717138602794,0.9431486969817574,0.5527266317191752,33.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.5,0,4912,0.6238631028428591,0.7880931819183284,,2.3916004540295117,0.22087226541385552,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.5,0,4912,1.0,0.9550846756392397,,0.9171396140749148,0.9556335519894886,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.5,1,4991,0.8574551609283632,0.9181571830954164,0.0417590350711488,1.0384306701178627,0.9175953769064493,30.0,0.13707165665869042,0.26162608181354435,0.08975229304523169,0.005710425755583177,0.1908083338612685,0.14071807031963213,0.02850241258829688,0.006108857635317433,0.04855675521604293,0.09310021520737087,0.009838403529826435,0.008003227812983403,0.03318310696534977,0.09594135178984331,2.953698250409181e-05,0.005363359522920284,0.00028642502752470076 +tHMM-discrete,0.5,0.5,1,4991,0.8469351902869701,0.9040485415602753,0.04276312105106271,0.9452354045574384,0.8879371930601048,24.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.5,1,4991,0.628061404462297,0.7901066039887634,,2.4069767441860463,0.8538227240003508,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.5,1,4991,1.0,0.9647975917120419,,0.9174418604651163,0.974712884607713,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.5,2,4969,0.8224196138847899,0.8864820546057344,0.047113738095202806,0.9780087208520353,0.8556609187168991,120.0,0.18129169775038306,0.47514285133149203,0.11677278219416012,0.007623440797183671,0.016508961473899245,0.1343645704910759,0.023573117298115953,0.007494081640825711,0.05500167582562766,0.09059081403087059,0.006070528488578534,2.8229173894032725e-05,0.014230474613996643,0.09349836446773059,0.0014268395836352428,0.005405109650582587,0.049130232670892504 +tHMM-discrete,0.5,0.5,2,4969,0.8049433476063902,0.8647706571261817,0.04892470953327467,0.9407528878187886,0.7868181186144002,38.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.5,2,4969,0.5732566339510452,0.6980389275687041,,2.3679745493107105,0.7599014468201132,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.5,2,4969,1.0,0.9455655693570788,,0.911983032873807,0.972962713884736,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.5,3,5083,0.912790040092804,0.9450565803875648,0.037069574093652385,1.0066158263309886,0.9576556098951019,33.0,0.07561532764743731,0.15133657484580607,0.08084173107697656,0.0036706442148529564,0.04946682384895654,0.14920354767088853,0.019313437307973615,0.0019892993374372374,0.04556411095454169,0.10821631495045776,0.004840340436146984,0.011807940745709622,0.011058539475292338,0.09177410493600688,1.1029048924764976e-05,0.005234484320299413,0.005464625741410446 +tHMM-discrete,0.5,0.5,3,5083,0.905558166483015,0.9361228794254028,0.04021682070886964,0.9579709311858411,0.9484249384310193,28.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.5,3,5083,0.7465991448750777,0.8369755387206274,,1.9058050383351588,0.7358878683156574,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.5,3,5083,1.0,0.9531841533285247,,0.9090909090909091,0.9879774966661098,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.5,4,4969,0.9104913515468878,0.9423843613909803,0.048315552538222374,1.0020747364182567,0.8747545682156067,42.0,0.07655850291975988,0.19570411797135678,0.07571814674955307,0.0030741744563094445,0.04253991157385367,0.1384911229786223,0.020741139842420982,0.006132216116520927,0.05508009605429807,0.09539536757531988,0.00021270871915281752,0.0032797213389090776,0.017410949486728017,0.0862798795885012,0.0037715694371911325,0.00010345946450453038,0.0025015518619807796 +tHMM-discrete,0.5,0.5,4,4969,0.9017750446397619,0.9186608903691272,0.0476487475210956,0.9715325189834368,0.7947899914821875,29.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.5,4,4969,0.7207849333054641,0.8523273695993205,,2.1003420752565565,0.6269635611657789,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.5,4,4969,1.0,0.9542226560235768,,0.9144811858608894,0.9202555539100199,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.25,0,2470,0.8046232401171961,0.866113193833533,0.1393144946049314,1.0513990474018065,0.7464230437722968,159.0,1.4673954507385296,5.14111840806845,0.31566990873147555,0.05463199236611232,0.162734629100739,0.15657101757199712,0.01901251603292217,0.013954870560765971,0.07130179475904787,0.11511390936924765,0.018540225640124017,3.9563547157706534e-05,4.505120607250099e-05,0.028664171020867657,0.23505802945133597,0.13173210184163667,0.04186784222118831 +tHMM-discrete,0.5,0.25,0,2470,0.79285694069065,0.7617669984801716,0.10716509068760426,1.0153599088945224,0.7243923314592879,178.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.25,0,2470,0.5713285477083415,0.6830998950845811,,1.7,0.1897766369504573,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.25,0,2470,1.0,0.9074987580341297,,0.8835820895522388,0.9316362828549893,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.25,1,2399,0.814036827521757,0.871557772450813,0.0540210409953173,1.0081952966896908,0.7531211869315768,55.0,0.12102774693902364,0.1338681832878923,0.11583311787072104,0.0035032065156404197,0.06641312453976976,0.15094096965269113,0.016252906846360896,0.004154886051664147,0.04405214686000126,0.12781548045242694,0.003000569367832257,0.004187582762179763,0.012005101031006778,0.07520420214874407,0.0005490369343665811,0.0005688825948522524,0.013744617713386028 +tHMM-discrete,0.5,0.25,1,2399,0.8081999115243095,0.8397220362813907,0.0548570204053977,0.9194467996096682,0.6507697760777367,31.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.25,1,2399,0.6076311419568817,0.6579926849734438,,1.9078260869565218,0.6025083410029438,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.25,1,2399,1.0,0.9056038302558556,,0.8626086956521739,0.857094942877925,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.25,2,2525,0.8024110317677401,0.8822191610910867,0.0652058218563806,1.0194189289024513,0.7377903497537983,83.0,0.25818051212265014,0.8187472812793268,0.15782712736688786,0.011662810866669614,0.06777495614429879,0.13771547202049303,0.019275141252290347,0.007819046397799517,0.039085913711317526,0.09303843965568794,0.017326755156231565,0.008029088371798686,0.002469245599726962,0.05907885383567456,0.01300865935685965,0.021769653077700832,0.04209367142498397 +tHMM-discrete,0.5,0.25,2,2525,0.7975667945416006,0.875460799686255,0.06822767039517355,0.9102199745936671,0.7113317977374302,44.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.25,2,2525,0.5871613920951054,0.7049547154145935,,2.0163934426229506,0.5765629495564154,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.25,2,2525,1.0,0.9434859203684265,,0.9052823315118397,0.9263711301759349,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.25,3,2538,0.9059412941339107,0.9232067662409872,0.05798771720765607,0.9992908668370587,0.9489253527698853,37.0,0.08778272408074597,0.057081675691977696,0.04848296448101916,0.006636269663595244,0.11149050059241306,0.14124758762509212,0.01627082832613685,0.005710902064197458,0.049337628588029286,0.09482161109316659,0.008025269868314281,3.7052154862350696e-05,0.028411926586370487,0.07435591001731127,2.0916417565988054e-05,0.022344923505335196,7.406781689028294e-05 +tHMM-discrete,0.5,0.25,3,2538,0.8886164790586782,0.8735370913950409,0.05735869334596286,0.9917083186293354,0.9379069464606813,24.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.25,3,2538,0.7636378915492833,0.841217650591774,,1.5462328767123288,0.7980642772500912,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.25,3,2538,1.0,0.9160711951544996,,0.8681506849315068,0.9304184467394481,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.25,4,2534,0.880589291348208,0.9243542125901213,0.06143129748997349,1.005304167961715,0.7882658316487858,104.0,0.11322740140556892,0.22751365676918162,0.06184347097446238,0.006040257876808307,0.068189981378237,0.14116304214428474,0.014595770857902106,0.005631157331158087,0.04869531484499783,0.10883937103490562,0.008952320062351611,0.0002488831993987431,0.01878286014292124,0.07408212655381481,0.007567947825135877,0.02506482282685995,0.00026667083081697885 +tHMM-discrete,0.5,0.25,4,2534,0.8662405114271945,0.886300556127134,0.05350907912900765,0.9509144753625101,0.6661056344509999,39.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.25,4,2534,0.7089280272833796,0.7798762398873005,,1.717607973421927,0.5089776719788348,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.25,4,2534,1.0,0.9430307499621534,,0.8953488372093024,0.7965441209561843,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.1,0,1058,0.7705119287870618,0.8435685823293385,0.09997817486183619,1.1475835418201883,0.24263272980940337,96.0,2.4619081991652867,7.62658977058892,0.4683733629596538,0.0778964144754442,0.0698390314862467,0.1668494152444082,0.02110290106482411,0.00046068892388444925,0.047618731873492165,0.1077557771147959,0.0014107991252381453,0.013463756804678791,0.00013096069837800254,0.21087847065582227,0.03684386072020566,0.0289942046647038,0.5397690412966418 +tHMM-discrete,0.5,0.1,0,1058,0.6419694263492337,0.5199964631447622,0.24242676957421386,1.3154705507761277,0.6119570508417226,110.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.1,0,1058,0.6103972477739875,0.6742404310314452,,1.4883040935672514,0.1557667739809048,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.1,0,1058,1.0,0.8868836042870093,,0.8450292397660819,0.879535142095998,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.1,1,957,0.7028367553743405,0.773100407192446,0.10589051564547074,0.9024350518512781,0.4319500326920389,67.0,0.4181744385623143,0.2660141826809247,0.3776597706447761,0.016425783038665286,0.11197795776823798,0.12133342742572059,0.04245756102052756,0.008337115396309555,0.024238492582131323,0.03739944985145093,0.020091578779209144,0.0001432101510494866,0.08842785815241871,0.06573656408738314,0.00010016189130677026,0.00022879125065243959,0.017564314253866037 +tHMM-discrete,0.5,0.1,1,957,0.6595308848383066,0.742511911776192,0.1339541007923043,0.7456181308225884,0.495045487154053,52.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.1,1,957,0.5790148984657908,0.6582177203762409,,1.4161490683229814,0.5390025004822978,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.1,1,957,1.0,0.8638403458239476,,0.8167701863354038,0.8968497560813873,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.1,2,1029,0.7420850181882066,0.8239040675172861,0.10497668106698911,0.9650304944092101,0.6000039740490447,75.0,0.38010548019794804,0.459877063689482,0.29867598700548237,0.010607709201992197,0.10911014668603876,0.12377645390471542,0.03489982342614063,0.011928134169404385,0.050637252727454535,0.10500419709009566,0.03667845645173971,0.010132061872753196,0.01601000972775106,0.00343253744506784,0.024669403318275576,0.003104026648223134,0.009332169246401735 +tHMM-discrete,0.5,0.1,2,1029,0.7168360367270704,0.6218071058786604,0.0978495198019034,0.9602026856804222,0.6683296736797558,67.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.1,2,1029,0.5456298868240254,0.5344347292329258,,1.496875,0.3037933633431617,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.1,2,1029,1.0,0.8892451633640748,,0.8625,0.9493886788861248,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.1,3,1079,0.8989005150938991,0.9084838955622117,0.055465590346491477,0.9685122287914241,0.8926989490464166,39.0,0.12202426700827938,0.11397771204654294,0.15691094723985458,0.0035476044355606724,0.25377934141191527,0.15607474351904188,0.024291712214874535,0.006075389385980552,0.07153870521128863,0.0999486864601335,0.00622100642262824,0.002227114825568218,0.021947069116633872,0.09246866073664277,0.01422865724415238,0.005639117809622757,0.0027950018838387384 +tHMM-discrete,0.5,0.1,3,1079,0.8845814164135094,0.8777058856106713,0.06158318521688407,0.9488054039018956,0.8823341673743591,20.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.1,3,1079,0.77496979394099,0.8131800006237228,,1.2317073170731707,0.8613844300738295,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.1,3,1079,1.0,0.8870813418855606,,0.823170731707317,0.9441973572150597,,,,,,,,,,,,,,,,,, +tHMM-ctmc,0.5,0.1,4,993,0.8482566196699981,0.8929106861501838,0.07376015522137076,0.8841946902802256,0.8948229067365276,39.0,0.2789851275910724,0.38903085046828834,0.2717991716526853,0.005238978622158899,0.2587576365196037,0.09608340888152286,0.012021706049997645,0.007697234453393555,0.06823332570404614,0.0774722466517455,0.001336109793955564,9.049504731244127e-05,0.017075010362617882,0.06897530413940592,3.543465729525408e-05,0.00012494891371154466,0.0001017006512320373 +tHMM-discrete,0.5,0.1,4,993,0.8449310899020095,0.8474036494388819,0.07257001327075457,0.8131599199778817,0.8264062523450336,31.0,,,,,,,,,,,,,,,,, +parsimony-kmeans,0.5,0.1,4,993,0.7320151568714443,0.7763966938221356,,1.353356890459364,0.47650636456018347,,,,,,,,,,,,,,,,,, +parsimony-oracle,0.5,0.1,4,993,1.0,0.8737512495874451,,0.8303886925795053,0.9060833916085875,,,,,,,,,,,,,,,,,, diff --git a/analysis/kptracer/results/state_table_anchored.csv b/analysis/kptracer/results/state_table_anchored.csv new file mode 100644 index 000000000..41b55f8b8 --- /dev/null +++ b/analysis/kptracer/results/state_table_anchored.csv @@ -0,0 +1,14 @@ +leiden,name,n_cells,exit_rate,thmm_stay_prob,fitch_stay_prob,published_scPlasticity,root_posterior,stationary,net_outflow_jumps,stay_lo,stay_hi,exit_rate_lo,exit_rate_hi,root_lo,root_hi,stay_rank_median +0,Endoderm-like,476,0.3811981394541146,0.7489095189543125,0.7192982456140351,0.2925386065208618,0.0009822314773366,0.0057258485444557,-63.10856736113965,0.23712497220216613,0.7764022841832798,0.28607214255205904,2.364450292108087,5.307361542507627e-09,0.10469312197509029,7.0 +1,Mesenchymal-2,8953,0.0200246937376611,0.9898741747132036,0.9883272767316374,0.0086902817246412,1.607794342841078e-06,0.0002212054569299,77.2418199489316,0.10610900079230243,0.9898951400942303,0.01998221746647613,0.878138879957243,1.444580277161603e-06,0.3606467068383123,3.0 +2,Mesenchymal-1,5527,0.0043140425676735,0.9974340143905136,0.9949433304272014,0.0022125129637983,0.0476176048427281,0.0009904508856369,-70.74808648050937,0.10427956393317463,0.9974340263115006,0.004312069732261748,0.8873649030433777,0.02659545589830781,0.21071718940475567,1.0 +3,Gastric-like,2747,0.3063170927006874,0.8091216013107277,0.7844611528822055,0.2763730348445457,0.026912658965507,0.078775698962893,297.5406433914329,0.5034444338605863,0.8723104877192477,0.18542491440963463,0.9532618706842635,0.00012618320826939151,0.23546561663517582,5.0 +4,AT2-like,3884,0.0437994214890174,0.9153769896660556,0.8894490457328051,0.1152851731352183,0.4673809039900731,0.6729993628342414,33.69726467602959,0.8839479055795366,0.942965448434696,0.021448463759417242,0.09336565087084987,0.041598487360267944,0.6992230648168251,3.0 +5,High plasticity,1465,0.6069110700234017,0.5803414339026265,0.5931800158604282,0.2504142174670855,0.0212180928695698,0.0531194440738875,-427.35920151722166,0.4274129188655374,0.6530716869705364,0.3837768801579809,1.1411565595260837,0.0001949376548798178,0.06464648454819244,9.0 +7,Early EMT-1,1010,0.3173291436813739,0.8041737025366862,0.8462633451957295,0.1894882024819766,0.0822067919438857,0.0155806783830013,112.20609747459838,0.5665919399623345,0.907373264276065,0.21314520733419803,0.5159412261510006,7.994459558480572e-05,0.26813525634429125,5.0 +8,Early gastric,1717,0.4014334281377704,0.7354096026809631,0.735207100591716,0.2893924500992587,0.2055566208015801,0.0096995949036688,663.9300753016175,0.39210376597929203,0.7462755982476319,0.2537010572751611,1.631281240698548,0.00024665161913839167,0.33834214787712347,7.0 +9,Pre-EMT,165,1.0702821091539625,0.4269139474878089,0.75,0.1555899246208989,0.068931783510153,0.0037375696203486,-44.68004429643099,0.001692478146725007,0.5032092774212589,0.8689317782996635,9.507998763567533,8.808481384723637e-05,0.1093317521800905,12.0 +10,Lung progenitor-like,880,0.0241130802661849,0.9750230898317565,0.9278096800656276,0.1577551474430397,1.339742055895246e-05,0.0562293408982815,-29.916355440239933,0.5483503705017986,0.998164180321817,0.0018978393480308404,0.3441051984193962,4.935968890219964e-08,0.0538917774745703,3.0 +12,Late Gastric,675,0.2282401198669378,0.8157801652350525,0.7551330798479088,0.2266387135169911,0.0002098163611091,0.0059042711066228,128.9382747878489,0.28026716499216764,0.8306383585928857,0.17365414383248295,0.4679588543294384,1.544852360508022e-09,0.3129101647401122,6.0 +13,Early EMT-2,32,1.2700276782346824,0.2666400554874334,0.7142857142857143,0.19662195653148,0.0629705418429352,0.0006660408426083,13.58831975801715,0.003704081627963142,0.5441923898872629,0.23245898481621827,8.125830460798657,0.0016030404561579543,0.242902283886602,12.5 +14,AT1-like,1408,0.3415447619403619,0.4736465542783793,0.7947761194029851,0.2063795839860826,0.01599794818022,0.0963504934874237,-691.3302402429342,0.2806937317789121,0.5715668798152577,0.22964495241701305,1.274779154402187,0.005124635689077206,0.1695179025150919,11.0 diff --git a/analysis/kptracer/results/state_table_anchored_delta0.1.csv b/analysis/kptracer/results/state_table_anchored_delta0.1.csv new file mode 100644 index 000000000..e58ed7bd3 --- /dev/null +++ b/analysis/kptracer/results/state_table_anchored_delta0.1.csv @@ -0,0 +1,14 @@ +leiden,name,n_cells,exit_rate,thmm_stay_prob,fitch_stay_prob,published_scPlasticity,root_posterior,stationary,net_outflow_jumps +0,Endoderm-like,476,2.369529713590103,0.6825092835492673,0.7192982456140351,0.29253860652086183,1.288324429702992e-05,0.00928856772319976,-2.214396470166321 +1,Mesenchymal-2,8953,0.09259290014031292,0.990004147094953,0.9883272767316374,0.008690281724641242,9.434963298968477e-05,9.556557675052822e-05,77.25090050424996 +2,Mesenchymal-1,5527,0.012211772082665643,0.9976401191383326,0.9949433304272014,0.0022125129637983387,0.04752571280339682,0.0007010794114584481,-71.31841424843815 +3,Gastric-like,2747,1.8331382262435236,0.7858931746600907,0.7844611528822055,0.27637303484454573,0.006942424808620194,0.10992269059646743,208.83324880241616 +4,AT2-like,3884,0.03156607333563252,0.9290241099669264,0.8894490457328051,0.11528517313521837,0.6075910016597587,0.5978153012708898,194.35439463584117 +5,High plasticity,1465,3.4803934219587735,0.5534750267885854,0.5931800158604282,0.2504142174670855,0.00703690233552353,0.06049574182733551,-362.3287475072311 +7,Early EMT-1,1010,1.590715894965446,0.7944073265943883,0.8462633451957295,0.18948820248197668,0.07152237193005659,0.025870698328454934,79.90252162842921 +8,Early gastric,1717,2.9862377807096987,0.6842529660775327,0.735207100591716,0.28939245009925874,0.0588544475830024,0.05061220484302111,506.23482424806343 +9,Pre-EMT,165,5.607851342398749,0.40952263982035936,0.75,0.15558992462089893,0.052942620594251553,0.0060059998564774885,-47.502944348176015 +10,Lung progenitor-like,880,0.03641113174380003,0.9763177504951854,0.9278096800656276,0.15775514744303973,2.9242204327416354e-05,0.055015520571913334,-25.08611004578941 +12,Late Gastric,675,1.4124232508056296,0.7682283890367712,0.7551330798479088,0.2266387135169911,8.917733100672652e-06,0.015157945699985161,15.410271631824116 +13,Early EMT-2,32,11.73115570274996,0.14860327477914562,0.7142857142857143,0.19662195653148007,0.08843619941845407,0.000656184016735517,46.806534229236206 +14,AT1-like,1408,2.3734299399914236,0.516392821988611,0.7947761194029851,0.20637958398608264,0.059002926052221474,0.06836250027731099,-620.342083060259 diff --git a/analysis/kptracer/results/state_table_anchored_delta2.csv b/analysis/kptracer/results/state_table_anchored_delta2.csv new file mode 100644 index 000000000..f105e27a1 --- /dev/null +++ b/analysis/kptracer/results/state_table_anchored_delta2.csv @@ -0,0 +1,14 @@ +leiden,name,n_cells,exit_rate,thmm_stay_prob,fitch_stay_prob,published_scPlasticity,root_posterior,stationary,net_outflow_jumps +0,Endoderm-like,476,0.104801352353862,0.7672337717833675,0.7192982456140351,0.29253860652086183,0.0004715879379356905,0.01053457852335749,-122.93022803494233 +1,Mesenchymal-2,8953,0.0050809152604459834,0.9898570915998458,0.9883272767316374,0.008690281724641242,1.117857051236333e-07,0.0005800411649226618,77.14701021270909 +2,Mesenchymal-1,5527,0.0012508666650640683,0.997391101435989,0.9949433304272014,0.0022125129637983387,0.047618945571678714,0.0022714426272858247,-70.56660890313803 +3,Gastric-like,2747,0.08655138933765089,0.8270060778540312,0.7844611528822055,0.27637303484454573,0.06980204080376992,0.08977331562056687,333.770337800054 +4,AT2-like,3884,0.020314088520534384,0.9235497510017153,0.8894490457328051,0.11528517313521837,0.5965696796943387,0.5550749920605503,114.19084690519878 +5,High plasticity,1465,0.1800830434570631,0.6159842728403773,0.5931800158604282,0.2504142174670855,0.0426045608492451,0.07381683503091913,-485.6319444702715 +7,Early EMT-1,1010,0.07331092616222563,0.8323838966789674,0.8462633451957295,0.18948820248197668,0.03400253825711751,0.013532116543786573,135.3709651546285 +8,Early gastric,1717,0.13244770573553485,0.7410104580211105,0.735207100591716,0.28939245009925874,0.0810214951963882,0.013880312339259444,637.3765029562719 +9,Pre-EMT,165,0.17751618747666992,0.5508617847537879,0.75,0.15558992462089893,0.0017201506625762588,0.004329777261057354,-53.98826649705296 +10,Lung progenitor-like,880,0.0061090154643024695,0.9846662284836865,0.9278096800656276,0.15775514744303973,8.81673287502135e-07,0.13100438938550446,-53.04018215877812 +12,Late Gastric,675,0.07722476673276675,0.8221935719700475,0.7551330798479088,0.2266387135169911,2.132517578305559e-05,0.00345700764744229,200.04494759783415 +13,Early EMT-2,32,0.17666130350496695,0.49976283148555084,0.7142857142857143,0.19662195653148007,0.11764232442363659,0.0006237428796883877,29.2085653289213 +14,AT1-like,1408,0.12701134399859584,0.5508923486070344,0.7947761194029851,0.20637958398608264,0.008524357968537602,0.10112144891565931,-740.9519458914348 diff --git a/analysis/kptracer/results/state_table_anchored_freeE.csv b/analysis/kptracer/results/state_table_anchored_freeE.csv new file mode 100644 index 000000000..ae384e035 --- /dev/null +++ b/analysis/kptracer/results/state_table_anchored_freeE.csv @@ -0,0 +1,14 @@ +leiden,name,n_cells,exit_rate,thmm_stay_prob,fitch_stay_prob,published_scPlasticity,root_posterior,stationary,net_outflow_jumps +0,Endoderm-like,476,0.08279206153016415,0.8226508485774253,0.7192982456140351,0.29253860652086183,0.0621657994974809,0.2525184521599021,-258.1558698172928 +1,Mesenchymal-2,8953,1.5882058938859753,0.4821969832623158,0.9883272767316374,0.008690281724641242,1.0547750591094587e-05,0.003820095095232094,4376.284455372418 +2,Mesenchymal-1,5527,0.006633751548308934,0.9960938432633227,0.9949433304272014,0.0022125129637983387,0.04759184640252881,0.14315365528525098,-49.39685879960187 +3,Gastric-like,2747,0.3634269175597668,0.7863612220698016,0.7844611528822055,0.27637303484454573,0.021538266260817375,0.06822091188064705,401.43138892433535 +4,AT2-like,3884,0.05163949217633169,0.8588430610273918,0.8894490457328051,0.11528517313521837,0.5653546402066291,0.302669604242106,267.74796525886643 +5,High plasticity,1465,0.36578089951285203,0.7428166990611552,0.5931800158604282,0.2504142174670855,0.09103697034403908,0.08629948409340443,-519.811108996467 +7,Early EMT-1,1010,0.0812982311107498,0.9131371589512656,0.8462633451957295,0.18948820248197668,0.15510387043350474,0.016094326590925835,92.92204758728035 +8,Early gastric,1717,0.8172239936929955,0.7214814984010118,0.735207100591716,0.28939245009925874,0.02531190510706778,0.03136219144306574,222.70238756407878 +9,Pre-EMT,165,0.33903627503980804,0.8633048026265002,0.75,0.15558992462089893,1.6653467577040106e-05,0.020692826188730053,-4326.884309071352 +10,Lung progenitor-like,880,0.01563725802899296,0.9843064685411186,0.9278096800656276,0.15775514744303973,3.0866198114739727e-06,0.045489938866527975,-13.9351943017225 +12,Late Gastric,675,0.02772299175546416,0.9700278155109228,0.7551330798479088,0.2266387135169911,8.950017026361011e-06,0.011470049965594297,33.34500796987735 +13,Early EMT-2,32,1.1919605852657962,0.2764660101644362,0.7142857142857143,0.19662195653148007,0.00937231612681196,0.0007272785265345357,-4.777794924660498 +14,AT1-like,1408,1.4503295770164648,0.49523754632598893,0.7947761194029851,0.20637958398608264,0.02248514776611426,0.017481185662079096,-221.47211676576137 diff --git a/analysis/kptracer/results/state_table_anchored_no3724.csv b/analysis/kptracer/results/state_table_anchored_no3724.csv new file mode 100644 index 000000000..2cde8a40a --- /dev/null +++ b/analysis/kptracer/results/state_table_anchored_no3724.csv @@ -0,0 +1,12 @@ +leiden,name,n_cells,exit_rate,thmm_stay_prob,fitch_stay_prob,published_scPlasticity,root_posterior,stationary,net_outflow_jumps +0,Endoderm-like,476,0.343152611210459,0.7610684935643127,0.7192982456140351,0.29253860652086183,0.001662033130298691,0.005977684086613672,-71.1880816982557 +3,Gastric-like,2747,0.3099093604809569,0.8086128520625293,0.7844611528822055,0.27637303484454573,0.027539074121118147,0.08009777217979666,292.21731024376504 +4,AT2-like,3884,0.043446953575945337,0.9155172065408573,0.8894490457328051,0.11528517313521837,0.4947628887034174,0.672081339781419,36.69043255918382 +5,High plasticity,1465,0.6168853720262566,0.578188377297487,0.5931800158604282,0.2504142174670855,0.024511749133300176,0.05312373059136363,-427.41312651094654 +7,Early EMT-1,1010,0.3356516707840706,0.8042315786063183,0.8462633451957295,0.18948820248197668,0.054080922505524584,0.016514674707840883,93.90214475229931 +8,Early gastric,1717,0.4018718083650065,0.7356575653229225,0.735207100591716,0.28939245009925874,0.20596902285347415,0.009855746735054242,662.3182337573521 +9,Pre-EMT,165,1.0456663159716943,0.4331158158235772,0.75,0.15558992462089893,0.04847257195729414,0.004010263305967586,-47.03800479490383 +10,Lung progenitor-like,880,0.02432386162742797,0.9747905212948879,0.9278096800656276,0.15775514744303973,1.684992397259355e-05,0.055830884552325134,-29.138740701722895 +12,Late Gastric,675,0.2106866078581282,0.8209639995035078,0.7551330798479088,0.2266387135169911,0.000310264473206835,0.005421887516040435,138.6148646289499 +13,Early EMT-2,32,2.7799136044435975,0.12901784471646235,0.7142857142857143,0.19662195653148007,0.12383372825735135,0.0005815879127037545,42.30988895986283 +14,AT1-like,1408,0.3368755730654401,0.476461500508638,0.7947761194029851,0.20637958398608264,0.018840894941041967,0.09650442863087483,-691.274921195584 diff --git a/analysis/kptracer/results/state_table_anchored_unit.csv b/analysis/kptracer/results/state_table_anchored_unit.csv new file mode 100644 index 000000000..d87f21b06 --- /dev/null +++ b/analysis/kptracer/results/state_table_anchored_unit.csv @@ -0,0 +1,14 @@ +leiden,name,n_cells,exit_rate,thmm_stay_prob,fitch_stay_prob,published_scPlasticity,root_posterior,stationary,net_outflow_jumps +0,Endoderm-like,476,0.2529849174540079,0.7603920334374508,0.7192982456140351,0.29253860652086183,2.632082720121107e-05,0.020700791284225695,-152.3120359082792 +1,Mesenchymal-2,8953,0.01020436884928656,0.9898603730424884,0.9883272767316374,0.008690281724641242,1.335074552867699e-08,0.0013066682431854282,77.07674942845176 +2,Mesenchymal-1,5527,0.0026330210470419417,0.9973845774334222,0.9949433304272014,0.0022125129637983387,0.04761903427623288,0.004877907089841931,-70.52859255333301 +3,Gastric-like,2747,0.19625315242440558,0.8365264486720321,0.7844611528822055,0.27637303484454573,0.0014839003115958115,0.16553749656999525,302.0111112766938 +4,AT2-like,3884,0.08126309570829014,0.9272525901840669,0.8894490457328051,0.11528517313521837,0.6392667533170393,0.28334545294871766,247.8912467137855 +5,High plasticity,1465,0.43336389993101415,0.6411877841467141,0.5931800158604282,0.2504142174670855,0.08597573232293974,0.12693422907738486,-550.7720958591102 +7,Early EMT-1,1010,0.1693079035333475,0.8493488295997709,0.8462633451957295,0.18948820248197668,0.00016715859823247946,0.03059152200011319,95.57422479965464 +8,Early gastric,1717,0.29523384412984854,0.7448050091761284,0.735207100591716,0.28939245009925874,0.04842024558335136,0.029514025339082067,612.2733961205518 +9,Pre-EMT,165,0.4967661697646583,0.573918989604708,0.75,0.15558992462089893,0.05566460994339856,0.007877517597217679,-17.926184593745973 +10,Lung progenitor-like,880,0.012432998179648236,0.987667517724953,0.9278096800656276,0.15775514744303973,3.2711520676857573e-07,0.2393652531869242,-61.278271759486074 +12,Late Gastric,675,0.2071411892221233,0.8156518627457684,0.7551330798479088,0.2266387135169911,2.0270541559224452e-07,0.00383922588830159,227.5360594590424 +13,Early EMT-2,32,0.6085479939201498,0.5337098169662419,0.7142857142857143,0.19662195653148007,0.09953905816566574,0.0008788280905280978,35.682439217868776 +14,AT1-like,1408,0.41924525624995795,0.5946916910785826,0.7947761194029851,0.20637958398608264,0.021836643482975,0.08523108268448254,-745.2280463420943 diff --git a/analysis/kptracer/results/transitions_fitch_anchored.csv b/analysis/kptracer/results/transitions_fitch_anchored.csv new file mode 100644 index 000000000..7a48a769a --- /dev/null +++ b/analysis/kptracer/results/transitions_fitch_anchored.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,0.0,0.0,5.0,0.0,7.0,0.0,1.0,2.0,2.0,30.0,0.0,1.0 +1,0.0,0.0,106.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +2,0.0,29.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +3,8.0,0.0,0.0,0.0,46.0,495.0,50.0,71.0,2.0,4.0,0.0,1.0,183.0 +4,10.0,0.0,0.0,122.0,0.0,233.0,30.0,38.0,6.0,52.0,1.0,7.0,422.0 +5,103.0,0.0,0.0,146.0,34.0,0.0,25.0,54.0,1.0,7.0,22.0,0.0,121.0 +7,8.0,0.0,0.0,15.0,1.0,42.0,0.0,0.0,126.0,2.0,0.0,20.0,2.0 +8,8.0,0.0,0.0,6.0,8.0,126.0,0.0,0.0,0.0,0.0,0.0,0.0,568.0 +9,1.0,0.0,0.0,0.0,0.0,4.0,2.0,0.0,0.0,3.0,0.0,0.0,0.0 +10,5.0,0.0,0.0,7.0,21.0,42.0,3.0,0.0,8.0,0.0,0.0,0.0,2.0 +12,254.0,0.0,0.0,41.0,1.0,26.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +13,0.0,0.0,0.0,0.0,0.0,1.0,3.0,0.0,0.0,0.0,0.0,0.0,0.0 +14,3.0,0.0,0.0,6.0,9.0,17.0,1.0,19.0,0.0,0.0,0.0,0.0,0.0 diff --git a/analysis/kptracer/results/transitions_fitch_anchored_delta0.1.csv b/analysis/kptracer/results/transitions_fitch_anchored_delta0.1.csv new file mode 100644 index 000000000..7a48a769a --- /dev/null +++ b/analysis/kptracer/results/transitions_fitch_anchored_delta0.1.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,0.0,0.0,5.0,0.0,7.0,0.0,1.0,2.0,2.0,30.0,0.0,1.0 +1,0.0,0.0,106.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +2,0.0,29.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +3,8.0,0.0,0.0,0.0,46.0,495.0,50.0,71.0,2.0,4.0,0.0,1.0,183.0 +4,10.0,0.0,0.0,122.0,0.0,233.0,30.0,38.0,6.0,52.0,1.0,7.0,422.0 +5,103.0,0.0,0.0,146.0,34.0,0.0,25.0,54.0,1.0,7.0,22.0,0.0,121.0 +7,8.0,0.0,0.0,15.0,1.0,42.0,0.0,0.0,126.0,2.0,0.0,20.0,2.0 +8,8.0,0.0,0.0,6.0,8.0,126.0,0.0,0.0,0.0,0.0,0.0,0.0,568.0 +9,1.0,0.0,0.0,0.0,0.0,4.0,2.0,0.0,0.0,3.0,0.0,0.0,0.0 +10,5.0,0.0,0.0,7.0,21.0,42.0,3.0,0.0,8.0,0.0,0.0,0.0,2.0 +12,254.0,0.0,0.0,41.0,1.0,26.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +13,0.0,0.0,0.0,0.0,0.0,1.0,3.0,0.0,0.0,0.0,0.0,0.0,0.0 +14,3.0,0.0,0.0,6.0,9.0,17.0,1.0,19.0,0.0,0.0,0.0,0.0,0.0 diff --git a/analysis/kptracer/results/transitions_fitch_anchored_delta2.csv b/analysis/kptracer/results/transitions_fitch_anchored_delta2.csv new file mode 100644 index 000000000..7a48a769a --- /dev/null +++ b/analysis/kptracer/results/transitions_fitch_anchored_delta2.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,0.0,0.0,5.0,0.0,7.0,0.0,1.0,2.0,2.0,30.0,0.0,1.0 +1,0.0,0.0,106.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +2,0.0,29.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +3,8.0,0.0,0.0,0.0,46.0,495.0,50.0,71.0,2.0,4.0,0.0,1.0,183.0 +4,10.0,0.0,0.0,122.0,0.0,233.0,30.0,38.0,6.0,52.0,1.0,7.0,422.0 +5,103.0,0.0,0.0,146.0,34.0,0.0,25.0,54.0,1.0,7.0,22.0,0.0,121.0 +7,8.0,0.0,0.0,15.0,1.0,42.0,0.0,0.0,126.0,2.0,0.0,20.0,2.0 +8,8.0,0.0,0.0,6.0,8.0,126.0,0.0,0.0,0.0,0.0,0.0,0.0,568.0 +9,1.0,0.0,0.0,0.0,0.0,4.0,2.0,0.0,0.0,3.0,0.0,0.0,0.0 +10,5.0,0.0,0.0,7.0,21.0,42.0,3.0,0.0,8.0,0.0,0.0,0.0,2.0 +12,254.0,0.0,0.0,41.0,1.0,26.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +13,0.0,0.0,0.0,0.0,0.0,1.0,3.0,0.0,0.0,0.0,0.0,0.0,0.0 +14,3.0,0.0,0.0,6.0,9.0,17.0,1.0,19.0,0.0,0.0,0.0,0.0,0.0 diff --git a/analysis/kptracer/results/transitions_fitch_anchored_freeE.csv b/analysis/kptracer/results/transitions_fitch_anchored_freeE.csv new file mode 100644 index 000000000..7a48a769a --- /dev/null +++ b/analysis/kptracer/results/transitions_fitch_anchored_freeE.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,0.0,0.0,5.0,0.0,7.0,0.0,1.0,2.0,2.0,30.0,0.0,1.0 +1,0.0,0.0,106.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +2,0.0,29.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +3,8.0,0.0,0.0,0.0,46.0,495.0,50.0,71.0,2.0,4.0,0.0,1.0,183.0 +4,10.0,0.0,0.0,122.0,0.0,233.0,30.0,38.0,6.0,52.0,1.0,7.0,422.0 +5,103.0,0.0,0.0,146.0,34.0,0.0,25.0,54.0,1.0,7.0,22.0,0.0,121.0 +7,8.0,0.0,0.0,15.0,1.0,42.0,0.0,0.0,126.0,2.0,0.0,20.0,2.0 +8,8.0,0.0,0.0,6.0,8.0,126.0,0.0,0.0,0.0,0.0,0.0,0.0,568.0 +9,1.0,0.0,0.0,0.0,0.0,4.0,2.0,0.0,0.0,3.0,0.0,0.0,0.0 +10,5.0,0.0,0.0,7.0,21.0,42.0,3.0,0.0,8.0,0.0,0.0,0.0,2.0 +12,254.0,0.0,0.0,41.0,1.0,26.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +13,0.0,0.0,0.0,0.0,0.0,1.0,3.0,0.0,0.0,0.0,0.0,0.0,0.0 +14,3.0,0.0,0.0,6.0,9.0,17.0,1.0,19.0,0.0,0.0,0.0,0.0,0.0 diff --git a/analysis/kptracer/results/transitions_fitch_anchored_no3724.csv b/analysis/kptracer/results/transitions_fitch_anchored_no3724.csv new file mode 100644 index 000000000..20b513540 --- /dev/null +++ b/analysis/kptracer/results/transitions_fitch_anchored_no3724.csv @@ -0,0 +1,12 @@ +,0,3,4,5,7,8,9,10,12,13,14 +0,0.0,5.0,0.0,7.0,0.0,1.0,2.0,2.0,30.0,0.0,1.0 +3,8.0,0.0,46.0,495.0,50.0,71.0,2.0,4.0,0.0,1.0,183.0 +4,10.0,122.0,0.0,233.0,30.0,38.0,6.0,52.0,1.0,7.0,422.0 +5,103.0,146.0,34.0,0.0,25.0,54.0,1.0,7.0,22.0,0.0,121.0 +7,8.0,15.0,1.0,42.0,0.0,0.0,126.0,2.0,0.0,20.0,2.0 +8,8.0,6.0,8.0,126.0,0.0,0.0,0.0,0.0,0.0,0.0,568.0 +9,1.0,0.0,0.0,4.0,2.0,0.0,0.0,3.0,0.0,0.0,0.0 +10,5.0,7.0,21.0,42.0,3.0,0.0,8.0,0.0,0.0,0.0,2.0 +12,254.0,41.0,1.0,26.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +13,0.0,0.0,0.0,1.0,3.0,0.0,0.0,0.0,0.0,0.0,0.0 +14,3.0,6.0,9.0,17.0,1.0,19.0,0.0,0.0,0.0,0.0,0.0 diff --git a/analysis/kptracer/results/transitions_fitch_anchored_unit.csv b/analysis/kptracer/results/transitions_fitch_anchored_unit.csv new file mode 100644 index 000000000..7a48a769a --- /dev/null +++ b/analysis/kptracer/results/transitions_fitch_anchored_unit.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,0.0,0.0,5.0,0.0,7.0,0.0,1.0,2.0,2.0,30.0,0.0,1.0 +1,0.0,0.0,106.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +2,0.0,29.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +3,8.0,0.0,0.0,0.0,46.0,495.0,50.0,71.0,2.0,4.0,0.0,1.0,183.0 +4,10.0,0.0,0.0,122.0,0.0,233.0,30.0,38.0,6.0,52.0,1.0,7.0,422.0 +5,103.0,0.0,0.0,146.0,34.0,0.0,25.0,54.0,1.0,7.0,22.0,0.0,121.0 +7,8.0,0.0,0.0,15.0,1.0,42.0,0.0,0.0,126.0,2.0,0.0,20.0,2.0 +8,8.0,0.0,0.0,6.0,8.0,126.0,0.0,0.0,0.0,0.0,0.0,0.0,568.0 +9,1.0,0.0,0.0,0.0,0.0,4.0,2.0,0.0,0.0,3.0,0.0,0.0,0.0 +10,5.0,0.0,0.0,7.0,21.0,42.0,3.0,0.0,8.0,0.0,0.0,0.0,2.0 +12,254.0,0.0,0.0,41.0,1.0,26.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0 +13,0.0,0.0,0.0,0.0,0.0,1.0,3.0,0.0,0.0,0.0,0.0,0.0,0.0 +14,3.0,0.0,0.0,6.0,9.0,17.0,1.0,19.0,0.0,0.0,0.0,0.0,0.0 diff --git a/analysis/kptracer/results/transitions_jumps_anchored.csv b/analysis/kptracer/results/transitions_jumps_anchored.csv new file mode 100644 index 000000000..638a6065a --- /dev/null +++ b/analysis/kptracer/results/transitions_jumps_anchored.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,5.315099327113812e-07,1.1879836411749084e-06,0.0014242844889125446,3.364692113331804,100.82099732715206,0.00011849510653308186,0.000747958204343894,0.00026485339182272575,0.0002140446376339636,164.5142146053141,0.00034481560984844485,0.012111856951286248 +1,1.872191517454666e-05,0.0,85.66658290448648,3.4197713512718444e-06,2.2517568935794253e-07,1.0733062353442825e-05,6.012182603341532,9.132568361501083e-07,0.0012594398119030823,0.4804056043403063,1.0789334156836976e-06,0.0001346527217979296,1.2543585610868782e-07 +2,1.4998642305358358e-05,14.918351182231527,0.0,1.2906519857385382e-06,4.7693260435970484e-08,1.4549689804094146e-05,5.82300392211549e-05,3.021452014687366e-07,1.5158727824652053e-05,6.447275030251078e-05,1.1082095019559805e-06,9.375696536507467e-06,1.8175382066555152e-07 +3,0.022229952360357217,2.738134840267819e-06,4.970903228255868e-06,0.0,0.0012702110955330075,1138.7669329749363,57.1940302706902,0.0015247552717143475,2.7458282953953415,0.0016730775861370129,0.00040691370342429143,0.0019007750578699176,123.79808156204494 +4,0.7110983691622907,1.7627118074986382e-06,1.9518000747474953e-06,0.0015900563932470292,0.0,145.7696625094948,0.0005447000206108992,0.004836329857214733,0.0010093546599952159,22.46357503650796,0.0008841107388845149,0.0009532166124826671,835.3007361546552 +5,32.26989453166403,4.019603008542922e-06,3.6301925490578197e-06,975.1180416812648,53.86347856268116,0.0,13.972732269881314,111.27571503681058,0.27267802907408106,22.760115972872843,0.0007110434420155522,0.0028951068623171983,115.25647104387188 +7,0.0005305396663792009,0.0002821389807267556,7.276531381272326e-06,0.6065580436458579,0.0017338797382040181,26.187640819461734,0.0,0.0005355601868905584,449.1511571151693,0.0010241591873158928,0.00016554101292717003,87.05711413499839,1.5933820942216224 +8,3.8370911165524553,5.013535250603664e-07,1.6932318757788771e-06,0.00034777443186438103,0.0002720969250907142,115.20677515419483,0.00010542188759247965,0.0,0.00013818243096271925,7.860133114936559,0.0002602989481151486,0.00020947101375796235,686.4966791214887 +9,0.000627020655977886,7.505085501607086e-05,6.616946184967626e-06,22.416507175439577,0.003980875441074388,42.53404378915752,322.95824165569815,0.0005746722915317798,0.0,10.424718396404348,0.00019308732401377264,9.164631179053325,0.002256074405360664 +10,0.00043413912906436314,1.9764533760665805e-06,1.5137771924219667e-06,0.017416706822372865,13.163456936237672,20.90598059637519,0.0006502123957474118,0.0005036518927436193,0.002321747529628488,0.0,9.512894844739392e-05,0.000592112425856147,0.002402050460582494 +12,293.4533305878584,2.579251028306789e-07,6.227741005249283e-07,0.0007649665767597055,0.00012480606348267933,0.0014481562788451383,4.700328764768385e-05,0.00018673526362013575,0.0001141467080990432,6.365878091649005e-05,0.0,0.00020480781828111075,0.00016929073093118594 +13,1.52474314912924,5.833536712672679e-05,2.728317187431979e-06,26.824306177381928,0.0578447559689595,29.125887571993477,52.25469906866044,0.0005162030274277347,0.010185094162857693,0.014994629960949227,0.00024842507344843465,0.0,0.004764778649333328 +14,0.003686308085872451,1.9781950800020214e-06,2.2817967823413283e-06,0.006281528878319359,900.100774366233,132.83254826364532,0.000623897193348506,38.186796527569705,0.0009284730412068415,0.003230044722523775,0.0009989105689489082,0.0009415118047614528,0.0 diff --git a/analysis/kptracer/results/transitions_jumps_anchored_delta0.1.csv b/analysis/kptracer/results/transitions_jumps_anchored_delta0.1.csv new file mode 100644 index 000000000..2e10acbf8 --- /dev/null +++ b/analysis/kptracer/results/transitions_jumps_anchored_delta0.1.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,1.3790167115155881e-06,3.640865847641174e-07,0.0016526486651216579,2.1170681490723817,27.192863438152138,0.00011143133027508014,0.0005865808216660819,0.00011798519307061841,5.757954182530534e-05,1046.516107152363,0.0003921309749466882,0.0016088588966975674 +1,1.7869879305467066e-05,0.0,85.37703401447969,1.7755622537621854e-06,5.21113345899987e-08,8.072171443058673e-06,5.43100883998633,4.533136745141474e-07,0.001465728804812507,0.49974696567156546,1.6049546863306643e-06,0.00033943493784227715,6.91858867627025e-08 +2,1.1370758646107979e-05,14.058491658205376,0.0,1.66662212927028e-06,3.004791911904825e-08,9.860574104191713e-06,4.922615680840467e-05,1.8565348424041422e-07,2.356914521421569e-05,4.815291007378266e-05,2.1825264736485956e-06,2.0547588970456896e-05,7.548906526602103e-08 +3,0.006186404904621843,8.951769634151944e-06,6.261225352350564e-06,0.0,0.006290913875271198,5001.128690089885,59.435231469949755,0.002903418114313865,1.9800318665765042,0.007091002405721977,0.0011984503468653242,0.01479140153635436,138.16378125883074 +4,0.16691742493996664,5.892983778412745e-06,2.7510846407711113e-06,0.00323826714106251,0.0,151.0117591200003,0.0005085231417422528,0.006917202327071216,0.0005426495724735504,14.791616716776264,0.002819071620550518,0.0009743402036505606,489.2562449880808 +5,9.849878201884149,9.613234161258762e-06,5.646885535411102e-06,4949.174143624951,108.30262026259376,0.0,16.810867616425146,66.93675335751762,4.2578155699555875,23.107256464285022,0.0015697525619558892,0.037714004157802926,13.597197591048689 +7,0.0003046906482330405,9.40275847441548e-05,5.061170520910904e-06,0.07250803234943928,0.0033277547216748284,11.197820615143899,0.0,0.000754697904578762,1578.0891852483628,0.0019652043658308755,0.00018517788461566728,354.63954744162885,0.0017024489673389492 +8,4.833034291082807,5.488645387187852e-06,4.230322824652341e-06,0.0019293236717606807,0.005585400901368342,234.2735015572003,0.0004103556223233938,0.0,0.000414249041898152,13.187636105575027,0.001009336470959142,0.0007374209698712129,4028.6616212199133 +9,0.00027419780797017574,4.5944269903082684e-05,4.554812867568376e-06,1.1524939370654457,0.008863873453607611,11.144148446389858,1506.4057109100372,0.0007716581770307018,0.0,7.821957124700001,0.0001868220567505336,10.31022029353692,0.0011403173182162366 +10,0.00022966920316275607,5.457840875240116e-06,1.5885143226327224e-06,0.00427595635871493,16.907190565368026,17.415251575017194,0.0018021758207446101,0.0006246976464493967,0.0027524099710725513,0.0,0.0001453639089418911,0.0017422493651006066,0.0009409188259689453 +12,1061.929117039917,1.289845661465627e-06,4.2491068020690616e-07,0.0013727447352913011,0.0003685714561874741,0.0025409626482866907,0.00010368060077419462,0.00045828427159381346,0.00010898926895682801,5.174936412209147e-05,0.0,0.00032731654388544565,0.000583680032006589 +13,1.2564773860502731,4.836978168718481e-05,3.7988329284837053e-06,41.49838699141019,0.0026327008270203432,91.91186880219388,276.0185628081523,0.00090036516862833,0.01578563967794637,0.0019739844285916962,0.0002588839470752829,0.0,1.1073503063224848 +14,0.0025156212045389977,6.303630939402365e-06,4.0777904479898485e-06,0.0029577184720970882,333.5332040376027,9.126116673355359,0.0005117350794477209,3707.7803938304387,0.0005185222316861788,0.0016716236059267438,0.0012793031282523695,0.00090922611266009,0.0 diff --git a/analysis/kptracer/results/transitions_jumps_anchored_delta2.csv b/analysis/kptracer/results/transitions_jumps_anchored_delta2.csv new file mode 100644 index 000000000..b1abc9e80 --- /dev/null +++ b/analysis/kptracer/results/transitions_jumps_anchored_delta2.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,2.926327579658628e-07,1.3257597569696247e-06,0.0016898959115035548,2.7913067638442293,129.51031064344784,0.00013670254227249123,0.0007794000734841865,0.0004117859234848688,0.0003312429771566796,40.80103062293366,0.00015917435440275757,0.22050975593518224 +1,1.9083730583270452e-05,0.0,85.71213533334803,4.775013585073869e-06,6.565710509595246e-07,1.3727861711081128e-05,6.1046571762824255,1.1747032967055395e-06,0.000883254163473491,0.47550813646537626,1.41606972910485e-06,8.941721678327894e-05,3.367074653456461e-07 +2,1.6102900249440403e-05,15.14539684803477,0.0,1.4089629045779273e-06,1.253255139375297e-07,1.577954784193188e-05,6.125814158903476e-05,3.602330468866406e-07,7.0602294778768365e-06,6.899461517494213e-05,1.2466554501120345e-06,5.37747081671218e-06,4.0926626657539336e-07 +3,0.016083516751105614,1.7006270785528707e-06,4.941052808780856e-06,0.0,0.00069532583100413,709.2067532608964,47.99622738354704,0.0012693005305612294,0.07197794610791672,0.0006319112413509348,0.000170664481538019,0.00038398344950061453,93.06961687050116 +4,3.978096855223038,7.036214033431297e-07,1.4345139013269822e-06,0.0013263684314653889,0.0,149.71319605216036,0.0006714899525180926,0.0017050084341050048,0.07249629253549951,24.26665032951822,0.00262110939761741,0.0003416623050011764,504.4693952345433 +5,45.85380017660211,4.450978144143131e-06,3.1906977848404577e-06,452.2764425356289,42.610674358850936,0.0,20.645800862101865,98.6758452165038,0.0025536708469183064,15.685301247394232,0.00047078796459818905,0.0005202777553304502,125.04014140558064 +7,0.0004586986452292123,0.0007543170373631668,1.170163799795179e-05,0.06822100949440417,0.0006683339999405815,32.04497756082189,0.0,0.00026014904844956407,209.12068050237107,0.00036231669375856937,0.00014825454759976605,29.24419654573019,0.49306103734339857 +8,3.1728595584352712,1.4172296640633427e-07,1.2894866717035997e-06,0.0002605774307930815,0.00018014288264179618,92.01881560911158,5.657844076897535e-05,0.0,0.00010064536646826355,7.0660239659363215,0.0001208480823680163,6.270808878526643e-05,643.6068092550762 +9,0.001319163645917733,0.00010438090335474841,1.875902787459969e-05,47.465690088429255,24.221653715179325,26.5973511943152,31.117864248739355,0.00021810091102897458,0.0,24.70580346772515,0.8873203947573829,0.28543932306832964,0.0017542118718907656 +10,0.0002959585232732476,1.471291523590994e-06,2.0441573557117015e-06,0.0007910220923758711,10.085604334551764,9.0736724645009,0.00029691643612058793,0.0004168258961036671,0.0006430826988378465,0.0,5.848964181637586e-05,0.0001462260204642849,0.007128614669651434 +12,241.73546220669957,1.2939228019913285e-07,6.84308371570717e-07,0.0009112032977056645,3.040130877958983e-05,0.0011640714213775603,1.991252147422529e-05,4.775733941974304e-05,0.00013185062699992359,5.3507899430983476e-05,0.0,9.966758266552602e-05,6.896680457792518e-05 +13,1.4935828466078849,3.917626306323668e-05,1.3301473175429257e-06,16.767843586317678,1.7541134494175366,8.977722466962483,29.736535412593728,0.00033350246987318176,0.0021076847105028365,0.006725472774930305,0.00015857860667725516,0.0,0.0011364424816717063 +14,0.004901473513838246,6.629197062260174e-07,1.8403832455770506e-06,0.010296533952976795,486.8507280276749,129.27950982012925,0.0005073314436629149,9.807911567645755,0.0008097700463503901,0.0017790160171793227,0.0009403482300761881,0.0002902573897781729,0.0 diff --git a/analysis/kptracer/results/transitions_jumps_anchored_freeE.csv b/analysis/kptracer/results/transitions_jumps_anchored_freeE.csv new file mode 100644 index 000000000..ee4a50389 --- /dev/null +++ b/analysis/kptracer/results/transitions_jumps_anchored_freeE.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,1.98514903074487e-06,1.0615542045896874e-06,0.001590347913635553,607.0432317195792,181.12816990155514,0.0022428097693693765,0.002425976475425155,2.3914564478317054e-06,25.07012555677705,0.0022440485137608488,0.002360938207340131,0.399236116554644 +1,8.206698596304703e-08,0.0,0.06402734223729398,9.80889162723782e-07,3.889172114275337e-09,2.4117668844323486e-06,0.0007380407655997592,1.4398723603004085e-08,4841.112575125588,2.7307104956019896e-05,9.008828216231753e-08,3.468144931700438e-05,1.8584449275459052e-08 +2,8.687822993433906e-08,0.016291894562846024,0.0,1.1348912517224753e-06,1.274247216982061e-08,1.0262635763239418e-05,3.363355333046897e-05,7.186119102344769e-08,22.829435968451456,3.696951117176456e-05,2.1849327139286714e-07,3.5187649948338467e-06,8.952285618071851e-08 +3,0.0017832826996891677,3.277055307741934e-06,7.403765971796874e-06,0.0,0.0025183288245525827,1217.8284736122168,25.586240279870132,0.001023443667567704,5.334757155272395e-06,0.00012654191872503142,0.0004023795987147602,0.0008052998107032165,3.991944259861624 +4,815.669966567676,1.1906382617149022e-06,2.6122463602665585e-07,0.0010655488602755382,0.0,42.03249792952706,0.00026914919359293524,0.0015894435007954512,1.4118886357247764e-06,0.0005125052245468039,0.0004379396168600281,0.0016418614110800698,18.70913378850591 +5,172.58014347177502,6.326669354978262e-06,3.290068263772566e-06,843.6823809605754,1.3784848514641963,0.0,18.126618394469592,28.180370694030994,1.158708119819976e-05,0.0004279311471509807,10.802911700257697,0.00222474240012812,0.0014208006614431592 +7,0.022233088579896726,7.282820491400252e-05,2.103714211296936e-06,2.2912307729384027,0.0003022207671923605,51.925185083234396,0.0,6.905615922125358e-05,0.00018974719158489878,4.682102622764566,9.800936738665565e-05,110.74336307840423,7.814719745348651e-05 +8,3.5386173039491102,3.0711149996745196e-07,3.3568909300759906e-05,0.0002403460220807326,0.0021366954249266027,28.61021365787335,4.6279729983232114e-05,0.0,6.653782352184177e-07,0.00018990768269651798,0.0001453713206373527,0.00026763638573633434,1140.1803040545615 +9,4.130134648831286e-07,464.87653627354393,72.17857974772512,1.2171462300576158e-06,1.5776819097272216e-08,5.064049966203747e-06,0.0014933536309777958,3.036397868625055e-08,0.0,0.0013341317694730685,2.687219042971367e-07,4.096703798964756e-05,3.8580715674525534e-08 +10,19.35727535272027,1.5708650081896536e-06,1.3491498222718434e-06,0.00026281092177158873,0.23814260026547013,0.002488646030670594,0.00021117752066198233,4.095633473635836e-05,2.012467239042094e-06,0.0,3.99363539585017e-05,0.0006971207256877487,5.460031764478348e-05 +12,0.00010173332143671681,2.9945813620538586e-06,1.6203182636835226e-06,0.0007289420480626733,2.0551305798538347e-05,40.37871965096009,0.0002611578211266973,3.231040083183527e-05,3.981848471200566e-06,2.075772857947881,0.0,1.6961497561211416,4.6333958130471906e-05 +13,58.83299038417381,3.1807493910232614e-05,1.4768347827502795e-06,0.004071518073453667,0.00033167951918448857,14.10394353260892,33.02467179828582,6.817237842792155e-05,7.194652879932703e-05,1.7036539769514336,0.00016048559273318277,0.0,7.352821678114674e-05 +14,1.8043909039437713,2.705344434376493e-07,1.3435968835775987e-05,0.0003699394317517329,0.003983658842142566,18.55640399460833,5.3096632905635975e-05,921.4441880606984,4.2007509257808736e-07,0.00010212659579366486,0.00039347283004990323,0.00027563060020992334,0.0 diff --git a/analysis/kptracer/results/transitions_jumps_anchored_no3724.csv b/analysis/kptracer/results/transitions_jumps_anchored_no3724.csv new file mode 100644 index 000000000..45c86f36e --- /dev/null +++ b/analysis/kptracer/results/transitions_jumps_anchored_no3724.csv @@ -0,0 +1,12 @@ +,0,3,4,5,7,8,9,10,12,13,14 +0,0.0,0.001722590630071293,3.481603539738451,104.53315254219014,0.00017204456087052463,0.0009583306284878633,0.0003359193867050881,0.00028122159810833197,136.09026824275105,0.0006438874703551154,0.23053449381474256 +3,0.007214734029730075,0.0,0.0015400425514041712,1152.0654270896791,57.13944497860968,0.0018389588760159476,2.2406084584882175,0.002040868180857775,0.00048781917665944544,0.0065151539128235425,127.9667888930667 +4,1.761550428264505,0.0018959315861426483,0.0,144.94405249117776,0.0006482253498311098,0.003043304481701419,0.0011805140629467868,22.567667921196364,0.0009906314281390607,0.0012336150656705532,827.5663685676598 +5,33.960360542128846,1002.4113841297386,56.59126151785396,0.0,13.815759451637318,110.3395115744608,0.2672899645366648,22.20912819943082,0.0008390644730963417,0.054241898913780284,104.29904064750895 +7,0.0009949442367747224,0.027935939639846464,0.002283008779474479,5.624995554205572,0.0,0.0005739193603321506,442.10054996124904,0.001227395026692436,0.0002136195843641804,143.965382566088,0.008286942160765018 +8,3.8645979361362466,0.0004168385278153521,0.00032739752655014566,114.54428956703912,0.00012328127935008103,0.0,0.00015984306865040102,7.871127413149274,0.00030582130110730685,0.0003667644137475252,687.0629700675039 +9,0.0011507162366825728,16.668725646239444,0.008837871896591971,35.2801274528667,308.7659646791559,0.0006790935685068252,0.0,10.91253433544149,0.0002463499576939758,25.946085475873485,0.001847102335899462 +10,0.0005500878345118436,0.00413128477630942,13.301553693441738,21.116336375003858,0.0007591576353798614,0.0006069611135406975,0.0023628370775768902,0.0,0.0001164995812495236,0.0009980313608499644,0.0028740116986496053 +12,274.7060557640313,0.000896825930811713,0.0001244127130881959,0.0017751773312780562,5.3842963212472324e-05,0.0002025384692815846,0.0001309557216892928,7.855752426202439e-05,0.0,0.0002875064587893173,0.00017753658068693485 +13,1.2197848803805256,28.094744161781186,0.002516727877309196,64.66685550580837,118.10663626335278,0.0006731434367188096,0.01050235748299661,0.001685950601725211,0.00027399892340621945,0.0,0.18329619531163865 +14,0.005494477745597132,0.0027434039560496905,886.7681508587104,128.58493174632736,0.0007371734872450635,40.67836334819818,0.0010827074016962063,0.0032577790969573446,0.0011764415976751258,0.001325325536325161,0.0 diff --git a/analysis/kptracer/results/transitions_jumps_anchored_unit.csv b/analysis/kptracer/results/transitions_jumps_anchored_unit.csv new file mode 100644 index 000000000..d56cace21 --- /dev/null +++ b/analysis/kptracer/results/transitions_jumps_anchored_unit.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,3.348777816437809e-07,1.4047611615426687e-06,0.0017274640300125267,3.1442391001813665,130.59028472011488,0.00011928561887974505,0.0008370753556874775,0.00045515892980398337,0.00041493350370848836,23.05619686524851,0.00016719522001174144,1.614706854523832 +1,1.9235578183717437e-05,0.0,85.71207039456785,5.044564472126518e-06,1.923121326017053e-06,1.7767635402038623e-05,6.076130528998825,1.2838621562436563e-06,0.0008774550405228989,0.47269799774442145,1.8649576791541323e-06,0.00011954499268117155,1.3230959269681232e-06 +2,1.6335669813529774e-05,15.183356482539033,0.0,1.526333393971689e-06,3.1543920077146234e-07,1.611868800831463e-05,6.187728886706565e-05,3.515786633437251e-07,8.724761958605946e-06,7.0775379221143e-05,1.563313166002674e-06,6.9882031986403275e-06,1.3427782718669017e-06 +3,0.019923143910759545,1.8450186351825024e-06,5.697714673717974e-06,0.0,0.0009966628048113167,604.7144386070228,36.43411687376163,0.0015923376258304932,0.006556002833311642,0.0003866958699553796,0.00014363521833966523,0.0003413312559742723,94.93018581861472 +4,5.695873139925336,5.13934652323501e-07,2.090393044297931e-06,0.0024503313417977898,0.0,129.2574584949056,0.19968167422325994,0.0009986134405070262,0.004353960107738695,30.30752098923268,0.001977200723255604,0.0003192104874163631,408.9744585426116 +5,49.05107902667267,6.9585755572241435e-06,3.2842184357359767e-06,367.8373998555938,15.352753734124429,0.0,29.402301320966274,92.00327706966772,0.001412028322476042,11.695581261257423,0.0005547431550526287,0.00043575555272306915,83.0999360725984 +7,0.00035273574312887806,0.0016026040044533364,1.5176105008499903e-05,0.002959786102537882,0.00020809437958683517,31.911053100863604,0.0,0.0001690127276119893,170.7823239890345,0.00023153828238194646,5.1164531217325485e-05,23.367023964679273,0.060771811994594575 +8,2.8638361389209996,1.2177774981521295e-07,1.170138997557557e-06,0.0002345360258426033,0.0001608094202892416,45.32141329939631,4.116208150181931e-05,0.0,9.599225601607683e-05,6.664908860705141,0.00011556672761208487,5.237154451675985e-05,662.3609340255829 +9,0.0008278923221087586,0.00016218927468442262,2.8338223664824602e-05,46.38746416800305,23.146414883122013,22.707324593311913,32.686152525869076,9.225961528797214e-05,0.0,26.939401769851017,1.0019720965290224,0.003532492817891553,0.0014749775973742442 +10,0.0002679972897422814,1.8132444508125476e-06,2.4780159518288757e-06,0.0003794636116383406,8.637075240583725,6.175450327841773,0.00023052144158200482,0.00039237514952280704,0.00043618046986433645,0.0,3.8058064180802756e-05,0.0001287042022061338,0.009281101256718874 +12,251.59527559869582,1.3813422108306367e-07,6.717109668664546e-07,0.0009374629731984596,2.022027445825565e-05,0.001039930537855522,1.4362272817013115e-05,3.249673628840254e-05,0.0001270304317164269,5.0353980296298215e-05,0.0,0.00010086963730607136,8.459064399730251e-05 +13,1.349463494237708,6.157036223415895e-05,2.095948049250096e-06,19.665960836035243,6.835773655765512,5.42813986394449,25.753062739006616,5.9441967810213944e-05,0.0039534041523447685,0.017740291265938107,0.00010876055716734681,0.0,0.0005597701465338631 +14,0.14425156167854697,3.639642295309688e-07,2.1535080096076453e-06,0.19805690034276208,269.4362034083247,223.11020014555257,0.000625307263896448,12.930945616298953,0.000432853942813803,0.0029505535852465316,0.0004627479613447225,0.00021827692766994872,0.0 diff --git a/analysis/kptracer/results/transitions_thmm_anchored.csv b/analysis/kptracer/results/transitions_thmm_anchored.csv new file mode 100644 index 000000000..32340bfe9 --- /dev/null +++ b/analysis/kptracer/results/transitions_thmm_anchored.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,6.372324342718492e-08,9.713412954704464e-07,6.6530027665593705,4.1677723972316745,69.41280533546265,0.03145137354114981,0.8601745564951884,0.006992322749508368,0.05372399221478722,87.99007158941305,0.001598833915739636,2.2927227796843592 +1,2.6184714929166115e-05,0.0,85.46297219380915,7.585588465027556e-07,7.581652389183308e-08,0.0003251468086328857,5.844857745025534,9.738828221745263e-09,0.16542690408738428,0.4819074715553403,9.106644285831332e-09,0.0010433200714362763,9.033162378163442e-08 +2,1.5560560048011263e-05,14.714330180999305,0.0,9.799341090323474e-09,6.945925045207987e-10,1.8541458005517583e-05,0.0004693572184283864,1.3799536491694775e-10,2.925962664680293e-07,8.509760835817846e-05,2.0323304729032947e-09,3.67847631266761e-08,5.877510345368723e-09 +3,4.024639781703874,1.0728153159475833e-06,6.734284103975648e-06,0.0,32.02273861728859,575.6431859285691,42.760272693008915,7.3664721534378,5.571159732355894,3.487818524035026,0.4878532477457761,0.47355928945757025,101.39039457156616 +4,2.4317244651904,7.368578080546312e-08,3.0562617828244607e-06,22.973231296681227,0.0,115.84030474138957,1.060620696931694,4.890813595029609,0.19859249343776458,22.39681005041576,0.2165108242779232,0.01006033106726827,402.38149410358915 +5,16.284565611381257,5.241513469539667e-07,2.7211172938842677e-06,364.27476088399646,54.852531426448486,0.0,12.633731671359376,53.50863799649444,1.2619584913192516,14.633110385690026,1.0406838535593381,0.16569957443752603,58.28993712961293 +7,0.27707942358252613,0.0003905564415135906,9.437614500028046e-06,30.09879103413956,5.242406758643755,40.622870139395346,0.0,0.4271750545586472,164.25541348876257,3.3264709413399345,0.023117227584941717,32.98593281327557,3.978865451051352 +8,3.841799434280909,2.495466650744563e-08,1.6430665215408437e-06,4.552821995552402,29.06947530201157,106.2089520691084,0.10401870411372911,0.0,0.021821356052803036,7.727724870686591,0.1511735968576397,0.004145066781188276,591.7004737465137 +9,0.0947975987722638,1.2799350528805022e-05,5.0892405105844605e-06,14.501483682889788,2.0219354666300338,14.596956289471121,86.20679451445872,0.16566454971746195,0.0,6.91809444175769,0.009683507508266014,2.9363777455850544,1.0227987166583785 +10,0.02390912980637599,9.895144131186353e-07,1.2212972482893941e-06,3.4897606943468436,12.2899674295792,14.19874698726623,0.038512663832525156,0.05754814829821364,0.008999948819568506,0.0,0.0017123592366265975,0.00029535603694499923,0.4445454832989684 +12,205.74551500094063,9.696369928625171e-08,6.275171546883781e-07,2.635157793886193,0.20414461738152467,9.90125912273626,0.0064777517469932445,0.19136757812092176,0.0023805524573609067,0.004726651828073541,0.0,0.0004640278918020277,0.2981269802692674 +13,1.1824828619112389,3.5217932485498226e-06,3.9888981907967104e-07,14.379188626629523,0.9508691802019467,11.295104413099947,19.952074315800854,0.10752167702357257,1.5662816255167973,0.13090102762216524,0.022154031177746836,0.0,0.5864893496619313 +14,0.6723292906262905,5.629971980412995e-08,1.4718359923850448e-06,12.12925941179339,397.88105977999714,46.58429307202435,0.3931433647531253,11.87695718930947,0.09562149031784317,1.3089823968222447,0.10838576539103224,0.005574876007185103,0.0 diff --git a/analysis/kptracer/results/transitions_thmm_anchored_delta0.1.csv b/analysis/kptracer/results/transitions_thmm_anchored_delta0.1.csv new file mode 100644 index 000000000..45b09701f --- /dev/null +++ b/analysis/kptracer/results/transitions_thmm_anchored_delta0.1.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,5.892749470404214e-08,1.1118627221295509e-07,2.445339093834855,1.9947209526610328,14.811305321343074,0.004357446334805664,0.040502728744383716,0.0013130427710394668,0.0011608167263647174,219.753987458586,0.00029821487925191004,0.30813765217742023 +1,2.3349198860997747e-05,0.0,84.8500950567573,4.9607935307786e-06,8.92100801036086e-07,0.00019368257117274436,5.281701474223783,2.706423309592698e-07,0.14863148102109808,0.5000087957880446,1.3223555041376185e-08,0.0009122127084561613,4.157411872150853e-07 +2,1.3480809664963186e-05,13.530514322829726,0.0,1.7465138037116332e-07,1.7302665677221177e-08,1.455055177399437e-05,0.0011123513697887707,7.61061759729064e-09,6.393896920490223e-07,6.436989663800281e-05,3.6894645705496777e-09,8.395046512656297e-08,1.8249000152824123e-08 +3,2.0492388577841365,1.3248549703986041e-06,9.858685532626243e-06,0.0,48.15502665831119,692.9695821459213,33.9590860490535,11.01337596130662,5.351728662563408,7.52067214396066,0.9493220816571795,0.9135591187089971,71.99258363380262 +4,1.1429655393480695,1.353986674234035e-07,5.219765851575346e-06,40.48601147904125,0.0,109.97719637186171,1.4584188410539973,38.46191989143704,0.4296862510054895,17.747455482226222,0.28294390296235905,0.01974381398708082,285.10777465276556 +5,4.466829027643593,5.104885305956362e-07,4.622788920575384e-06,517.8140814908082,47.765849443841795,0.0,17.52394452694647,25.05822583958563,2.6967496565140365,10.732190542447732,0.6848511209644607,0.47282600383233125,23.950992056641606 +7,0.11725148166242022,0.0001411990353291978,7.92350718651841e-06,38.054304556373694,6.2471047406718965,35.08230035147521,0.0,0.7625241681850262,165.72630470788974,3.6992555865152723,0.025131778785954254,35.202031481539734,2.280883040262224 +8,2.7820360608976267,8.5880342176949e-08,3.9009205470618e-06,16.211392589470762,63.14481076668588,91.5836062439035,0.36509201608903397,0.0,0.09226613839691569,8.511896017889681,0.3437281449562325,0.010802755581147318,644.3285035559733 +9,0.018543702416378434,1.1695656486348407e-05,1.988895008302883e-06,8.53759994111776,1.6324196625792993,7.800879503037078,103.30198316986456,0.2079679201854814,0.0,3.848193126052099,0.005752501093237445,3.1973242060790197,0.5626514792086977 +10,0.003618417445102431,1.0952381287296165e-06,9.687818297715142e-07,4.921623522697584,14.797270917781924,9.068163060355442,0.08791949770646378,0.03892529237264412,0.03463955767867423,0.0,0.0016165380014836273,0.0006336100345678412,0.24273205709771958 +12,229.20347544079024,8.241269402562378e-08,3.1251002916684615e-07,2.903917572822223,0.2711540066262773,5.165726839916335,0.004581286170298958,0.03943506577707678,0.0023811121219352695,0.0013169695459035076,0.0,0.00024480274410521366,0.22347062138166854 +13,1.027857008002822,1.4804743117357294e-06,2.7018804081570797e-07,16.774349678477442,0.7997550085658519,19.823354563396943,44.87621318475273,0.1433126319710651,2.0245142723841707,0.11318051970856838,0.01283592493598137,0.0,1.0415512043858903 +14,0.7636670023428097,1.0932375790613765e-07,4.034752735344703e-06,17.892312634113342,115.95161387789047,27.21296971539632,0.4303095439038178,245.37312425076854,0.10805772261901625,1.6078602102233412,0.34526301213477395,0.012015213963634325,0.0 diff --git a/analysis/kptracer/results/transitions_thmm_anchored_delta2.csv b/analysis/kptracer/results/transitions_thmm_anchored_delta2.csv new file mode 100644 index 000000000..f0edd2e6f --- /dev/null +++ b/analysis/kptracer/results/transitions_thmm_anchored_delta2.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,5.415458754291761e-08,1.126200626117376e-06,6.644767594899335,4.17305447719831,93.50478033615329,0.0643362655440843,1.2841264171949154,0.006941050612715166,0.10321182125621724,27.96342045316968,0.0004779591823831853,4.34057873493416 +1,3.153823786924938e-05,0.0,85.52824799182096,4.834927940685753e-07,1.999218779573486e-08,0.0005306836928031771,5.937959041473115,2.830711171688561e-10,0.16020750311943863,0.4813324308337898,1.7897809640688453e-08,0.0007617856861515437,2.613871203637854e-08 +2,1.618920428191232e-05,14.96116198060761,0.0,6.855282534284909e-09,4.789046319042738e-10,2.00204107960928e-05,0.0003949246976522042,8.300684666891257e-11,2.9469224326665993e-07,9.209597186338733e-05,1.852809588811715e-09,4.104290764650226e-08,4.440139807930116e-09 +3,4.825066869352998,8.393397688938415e-07,6.023279149868228e-06,0.0,12.330962932929246,524.1371739083494,44.4806116064928,6.828680127653639,2.3701803107131587,1.0702979924163194,0.02703371395906211,0.08726625039224298,97.44889211750042 +4,5.647427270521817,3.2397102528494635e-08,1.9215261447343777e-06,13.673085071569101,0.0,132.7715752346342,1.1121119690485635,2.982955078802073,0.12753005914205667,24.596550686455583,0.15117347710329174,0.0014510412971849584,349.2197902680772 +5,27.95075517590702,5.151579878165822e-07,2.1380821958175077e-06,262.7759299080843,45.372070220358125,0.0,16.030263551375516,63.19878456391219,0.57833792171481,12.390690016791602,0.2718081497474583,0.040023413380608684,70.63678510241924 +7,0.17848500123499114,0.0008905129755962363,1.2844084542189268e-05,16.450198706756463,6.226913873162406,35.76361228801658,0.0,0.09398686803306568,158.54982904406805,2.540701729894063,0.03485469789577578,21.38648335348943,2.2391613296169135 +8,3.8123011364507757,7.865267044499877e-09,1.4149144283979766e-06,1.9048996208140756,12.572172478873616,98.63739173426107,0.04089587787814404,0.0,0.0021480558688743703,7.2491879406675475,0.006450674129554177,0.00029687370881852163,594.2865687256531 +9,0.19762886703218246,4.265979104566672e-06,1.7039681776896485e-05,32.978513996158306,19.345917560873424,15.399337584611636,17.889854913375096,0.03726094941243024,0.0,21.879922269149432,0.8197820372972874,0.18154245037000347,0.787452359061989 +10,0.013212081083635576,5.78767125334789e-07,1.647249437651906e-06,0.48371334436753827,9.394874904877387,7.696529071397752,0.01080996501102571,0.03387627025760033,0.0018440713684080888,0.0,3.416313995054298e-05,3.6534303234295644e-05,0.34636901938189923 +12,216.0377786910933,1.1000563937343598e-07,7.666732207416088e-07,1.4659343317504796,0.015135674602481594,11.685505327442845,0.0006176866164484238,0.015769591819347728,0.0006169961938083746,0.003035428718992795,0.0,7.988116741374117e-05,0.12191561401135852 +13,1.2847658222941183,2.3870546759484342e-06,3.059952737989327e-07,14.48062459390732,2.8680312440802322,7.674172624999035,22.1987447696568,0.07425882303999852,1.684438914633338,0.0639618924455638,0.0021246067754607883,0.0,0.5769220573368096 +14,1.0684556830302854,2.5657236665786252e-08,1.243966694084282e-06,8.978167233668152,303.79367181794896,57.606766333233175,0.3275645234303879,6.586112894324693,0.023426567928525193,0.6424995053809227,0.024760509292801255,0.001063129277140786,0.0 diff --git a/analysis/kptracer/results/transitions_thmm_anchored_freeE.csv b/analysis/kptracer/results/transitions_thmm_anchored_freeE.csv new file mode 100644 index 000000000..198ac7782 --- /dev/null +++ b/analysis/kptracer/results/transitions_thmm_anchored_freeE.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,3.136568771695197e-10,1.3906990958059513e-06,15.100244694373016,399.05950578316947,115.08956871516199,1.311496306528087,0.5284027773887127,5.227729346707558e-06,21.820828030572816,0.348210583058384,0.01749155792082333,1.2952150285619288 +1,1.1469181555752939e-07,0.0,50.36677273689612,3.817264120387568e-09,6.136019572668326e-11,3.507379337331947e-06,0.0016615714772128833,2.8292100857194548e-11,4379.613092545594,3.577129068545695e-05,1.0765468436747773e-07,3.289557881772459e-07,4.026360088977754e-11 +2,1.1268481322171158e-09,2.1941712323648024,0.0,4.869583804820139e-09,1.6712300069898748e-10,1.2059146767516432e-05,3.4383859735749334e-05,2.1576838933867756e-11,20.177503756239773,3.4604528151925056e-05,1.2288062472809363e-07,6.663046562243564e-08,1.3078233127696534e-10 +3,24.262916137386636,1.693496735184568e-09,9.813137665481857e-06,0.0,2.391297935474752,738.7186267817152,22.955170809424903,1.5418488405137984,8.075055775999231e-06,0.015077029407372792,0.45633573051653564,0.10618284941950294,3.817165067661767 +4,603.6705134448832,1.769726083134037e-10,2.8183308873802e-07,5.163363833989835,0.0,53.49419417451108,0.13083852703566817,1.686261351597795,2.4950127130462744e-06,1.75662469857326,0.053085719058681535,0.009091738456679592,15.86503252451408 +5,102.68262141598551,1.5072013525232773e-07,1.313062102500332e-06,359.6421964148761,8.430323803317771,0.0,14.777315350922223,22.03185224353028,1.5491816822313802e-05,0.10332784050153668,9.225710428261847,0.24239524990785677,1.3982374660184167 +7,40.292583318387024,8.027003727851024e-08,2.0324224357429076e-06,9.380220655988065,3.0096265721089583,45.56058355677589,0.0,0.01625537157022251,0.0003540682147968194,5.14429360692526,0.007888887819355572,39.43125622147309,0.01412837950165549 +8,4.118430424188516,2.0967074541031935e-10,4.5823441665949824e-05,0.22763501964796581,0.09852772981799558,29.84053433274665,0.002579709771867088,0.0,1.0199443135188995e-07,0.0007570169413316221,0.003625785802979194,9.900367073765047e-05,538.9804069644371 +9,7.204330899737403e-09,51.50293984761109,21.40177625084353,1.075501893294642e-09,2.067141338583385e-10,3.554356234351526e-06,0.0006448796516572335,5.2091021077495145e-11,0.0,0.0013318779654010477,1.018938416834968e-08,1.982266093508696e-06,5.3633987790557e-11 +10,17.165170919091306,6.880423778723025e-11,1.3366184512139374e-06,0.0034913479922268507,0.7027300587727149,0.18039871372925068,0.0004164434381453953,1.6090786928046573e-05,2.4496023137854816e-06,0.0,1.4562057333727628e-05,0.00010598980314418643,6.997781543184904e-05 +12,0.2920599717237748,1.5871359196295024e-10,1.6742569794549407e-06,2.429618839298549,0.002131651494317534,37.09307963732854,0.08778268135657084,0.0017973072692133625,6.950658420991556e-06,2.071298944240711,0.0,1.4690315925292308,0.0023120105103304036 +13,18.66112593956619,1.7904187251338605e-09,6.473459201291577e-07,0.6434347371488361,0.34509448072027205,5.103914201969495,10.665719280436706,0.0010047980204333303,1.627213686785637e-05,1.0738873710451566,0.0031094143306956996,0.0,0.0006255447296921699 +14,1.581418218531271,1.3941626950791617e-10,1.7310123254739716e-06,0.2430445939957096,0.04180551546775362,13.264186930563744,0.0014852202726672572,324.76281556781606,4.872189981745036e-08,0.00011539950676727502,0.006131939317787857,7.103286850838735e-05,0.0 diff --git a/analysis/kptracer/results/transitions_thmm_anchored_no3724.csv b/analysis/kptracer/results/transitions_thmm_anchored_no3724.csv new file mode 100644 index 000000000..69086eb15 --- /dev/null +++ b/analysis/kptracer/results/transitions_thmm_anchored_no3724.csv @@ -0,0 +1,12 @@ +,0,3,4,5,7,8,9,10,12,13,14 +0,0.0,7.181227938886856,4.424708910622154,71.30608410275457,0.04194968501829936,0.9677735414153239,0.008318298059597213,0.06216995511209923,74.84214208897609,0.002245274896464945,2.464521778133494 +3,4.247191643657722,0.0,32.44084607610779,575.7921575092488,42.324874494160966,7.362344537162893,5.437795519408376,3.4320650458606137,0.44913360920851714,0.7567634126809195,102.22505063710757 +4,3.140077528799653,23.12585821569985,0.0,115.24490254463291,1.0703270303105032,4.982286477189283,0.19145322071187562,22.413050852535033,0.24122928852116862,0.009076619766060516,401.33540470963015 +5,16.952376077511836,370.9333081748819,54.67644351883084,0.0,12.64982926898705,53.0519407270386,1.2703595670100438,14.321491546222447,0.9348783866056625,0.2532524523920959,55.5165238808527 +7,0.3291818256168209,29.396635863948184,4.686883115704922,34.959248916841304,0.0,0.39804128977814,163.61021164537183,3.438938065472128,0.019567453839721378,36.25239383197685,2.471843000194307 +8,3.8854974077694306,4.509732423517818,28.439189275475275,105.49402260536021,0.10097602241107696,0.0,0.020673294478947068,7.719064345503524,0.13167359622541305,0.003675583659648667,591.7248398074242 +9,0.1157463524847912,13.028530095896395,1.7964658997573444,13.773452577146502,85.07067481341775,0.14166429792982985,0.0,7.179768606815626,0.007923141759284541,3.8146550307361418,0.8471848295880535 +10,0.026824953264480814,3.565111749348271,12.42134758351762,14.307873378507846,0.038163610337785106,0.05793680701773626,0.008876022937625327,0.0,0.0017280360799848105,0.0004044612737310318,0.42931887945907576 +12,202.07358804967822,2.6961861290896665,0.18987454680308924,9.93579396140121,0.00690268024940945,0.17473561954602518,0.0024298908558971727,0.004718436954562709,0.0,0.0004295600749029822,0.2723810274969175 +13,1.0184755982536797,15.791184632918906,1.1472791038379138,21.458941285732013,39.966460673776076,0.15935201717669065,2.1729159240235463,0.15047051953633916,0.01822201968706022,0.0,1.5246252778918492 +14,0.7002638350941833,12.02313701664656,394.8401958979583,45.70105322965451,0.390641977773305,12.415035290219583,0.09103705757962313,1.2745888094570654,0.0956776522969756,0.005141865515979083,0.0 diff --git a/analysis/kptracer/results/transitions_thmm_anchored_unit.csv b/analysis/kptracer/results/transitions_thmm_anchored_unit.csv new file mode 100644 index 000000000..8c8023218 --- /dev/null +++ b/analysis/kptracer/results/transitions_thmm_anchored_unit.csv @@ -0,0 +1,14 @@ +,0,1,2,3,4,5,7,8,9,10,12,13,14 +0,0.0,6.019651037923613e-08,1.2283063665797512e-06,6.751147111479019,3.8725126648333728,97.52028950588219,0.06577713999109355,1.1769547354378929,0.011765838001857072,0.13077923836472047,16.497003431730366,0.00026831927345504544,5.0933114502182 +1,4.24211585460817e-05,0.0,85.5280641988367,7.366581439706419e-07,2.618921092522704e-08,0.0006985181637805915,5.894206058963257,3.393731377059946e-10,0.17124324976643154,0.48257894783730954,3.273504756288554e-08,0.0007615246622148096,2.3560923348371714e-08 +2,1.6422905870431957e-05,14.999016002369613,0.0,7.11385907087187e-09,4.760874612677503e-10,2.1841101593440928e-05,0.00038336783596640414,7.979340736937175e-11,2.9214211114364943e-07,9.524147049599005e-05,1.9182251081467443e-09,3.888260426709993e-08,3.865173086840131e-09 +3,5.613531413837491,8.19937355109808e-07,6.801477195775433e-06,0.0,6.3755972963270615,484.98804322942925,37.51711108409191,8.132864245425973,1.578572680491406,0.584313797106055,0.006602146760881136,0.01682378413354151,94.18240908327479 +4,7.0390067505147815,2.7785672935924846e-08,2.454572199670749e-06,9.53757258944451,0.0,139.94278285155235,1.4908585108165302,2.552183006198576,0.043273545835794756,30.235349910864564,0.08229348881794629,0.0006588696655750013,332.5702180466586 +5,32.10012903740189,5.44789009349879e-07,1.974629610546079e-06,238.57874357098953,18.42606503916116,0.0,22.275753616721108,64.46511358959651,0.5414108539242007,9.906789877326027,0.19862914284732952,0.02186277149099082,51.71977682994109 +7,0.11881280019963263,0.0018229749962910124,1.6767288867750047e-05,9.097844661698842,0.8045338019042128,33.66805892690409,0.0,0.006911085072489336,143.4153246657153,1.1950309706776792,0.003941186866957303,18.587516577623465,1.3412057018387171 +8,3.779086632774359,1.2571138346234356e-09,1.4445854690543592e-06,1.110836994883587,7.051927801531464,92.36031065121213,0.010741251198885068,0.0,0.00018976210872661607,6.924852012402085,0.000959307353772055,8.063455001405704e-06,588.0217752022452 +9,0.178453703705034,3.936209443436234e-06,2.609102585730087e-05,38.47932354757178,22.10179591479443,17.662087104393777,23.222966399289184,0.01207771406202696,0.0,25.630563308576654,0.9560989780775471,0.03340207347237652,0.479332703798213 +10,0.011524695832693234,4.5053977186475386e-07,1.9633807526321345e-06,0.13961486934018574,8.133067726680551,5.692138065239282,0.009231913008334763,0.03394905805343361,0.0007789093548162912,0.0,3.0904844855887525e-06,3.8149801969949904e-05,0.3109431973305821 +12,231.75187598197766,1.08040299308111e-07,7.795462237062451e-07,0.9593746050955644,0.004647793282189316,12.531080790043026,0.00017078001338514758,0.009703189395995992,0.0005149212882830768,0.003318473257414896,0.0,8.12778777424409e-05,0.029360928300098798 +13,1.288191134550071,1.3761914871589669e-06,6.693865924677962e-07,17.68768650594592,6.715255924498759,5.818194001677795,21.763501816351393,0.002458234744266933,0.9123316792761562,0.07506411891775185,0.00019499192692704886,0.0,0.0814447597666476 +14,1.5511756371334888,8.107012030120005e-09,1.4004581020359847e-06,14.642619905377616,202.11754934926452,98.82266722233179,0.4160933828509836,10.595078146049469,0.006909670816268825,0.4408279517335718,0.00834436955690884,0.00046454502563641006,0.0 diff --git a/analysis/kptracer/results/tumor_benchmarks_anchored.csv b/analysis/kptracer/results/tumor_benchmarks_anchored.csv new file mode 100644 index 000000000..9c611eceb --- /dev/null +++ b/analysis/kptracer/results/tumor_benchmarks_anchored.csv @@ -0,0 +1,22 @@ +tumor,n_cells,n_edges,n_clusters,fitch_changes,fitch_plasticity,thmm_changes,thmm_plasticity,thmm_jumps,path_autocorr_leiden,path_autocorr_z_leiden,path_autocorr_thmm,published_mean_scPlasticity,cell_spearman_vs_published +3430_NT_T2,502,641,4,48,0.0748829953198128,86.82633057778213,0.13545449388109537,196.75901272627095,0.008236228719144442,6.039748163390079,0.002715937585085175,0.07210215414957862,-0.12768497247593183 +3432_NT_T1,219,355,6,86,0.24225352112676057,71.0224360135352,0.20006320003812733,128.40101734499865,0.10966428215135356,24.65355406744032,0.19162725283533677,0.23482823540233014,0.73640820328226 +3432_NT_T2,196,294,6,71,0.24149659863945577,66.69163339349181,0.2268422904540538,91.56460446598135,0.11417602954893143,23.783519419233873,0.1629188562994875,0.24251616347783578,0.8834919097669092 +3433_NT_T2,327,417,6,146,0.3501199040767386,135.5630465914531,0.32509123882842467,174.87378326867497,0.030279669808902003,9.94760141810501,0.03533883384454383,0.279334560530666,0.4252560581651581 +3434_NT_T2,1103,1562,5,366,0.23431498079385404,313.84213125473667,0.20092325944605421,510.89599991572493,0.007061926992934453,15.101117748645624,0.01765250569719532,0.18559919817458384,0.06618621751729258 +3434_NT_T3,2220,2688,3,695,0.2585565476190476,666.1645512701557,0.24782907413324246,697.6458846629939,0.0005346996401901756,3.247276721845305,0.0007043381489655953,0.18281316496814853,0.09515369458064488 +3435_NT_T1,1069,1829,4,212,0.11591033351558229,202.15512089880767,0.11052767681728139,396.20701925752746,0.07149819588018567,114.64692541084038,0.11059510509759381,0.11283396391171881,0.4235750640890932 +3435_NT_T3,316,578,4,79,0.13667820069204153,106.8864094031602,0.1849245837424917,218.78425163372623,0.013744786038700916,7.44761668431199,0.020067992147329714,0.13783076096852143,0.6841912746691071 +3435_NT_T4,272,402,3,31,0.07711442786069651,46.93291933458659,0.11674855555867311,90.21086039301068,0.0027370681606747505,2.5350750977635665,0.04106248327168731,0.07552029526510093,0.6503281132533111 +3435_NT_T6,168,283,4,29,0.10247349823321555,36.13772033921417,0.1276951248735483,67.29977034158667,0.009869623435229015,3.026562142481424,0.10203821372237794,0.09658270906188489,0.8308094674879308 +3513_NT_T2,2053,2693,4,329,0.12216858522094319,433.6691529702424,0.16103570477914683,875.3554632714297,0.013945027872155841,29.52369263242612,0.01615988174861471,0.09632015926093254,0.2912738770578774 +3513_NT_T3,1435,1782,5,540,0.30303030303030304,466.70720107225594,0.26190078623583385,768.0413759809588,0.04039735001696451,37.04850556282406,0.04877743236493868,0.28500866882729725,-0.015161257583754886 +3513_NT_T4,594,816,5,271,0.3321078431372549,277.97219711763347,0.340652202350041,582.5035832365937,0.03097684660967981,16.717879854265348,0.04366793830954744,0.2856110572553209,-0.1386551188165085 +3513_NT_T5,342,518,5,150,0.28957528957528955,169.46105568875265,0.32714489515203216,277.4115161794326,0.03260614721173709,15.953804269700129,0.05253377823218697,0.26264714581399473,0.450233985198588 +3520_NT_T1,148,281,4,30,0.10676156583629894,46.54295303556737,0.16563328482408318,92.87985550610969,0.03705425172983887,7.110870964981587,0.07665930417022658,0.11786291073108833,0.807079714292386 +3703_NT_T2,105,187,6,52,0.27807486631016043,76.0757267758947,0.4068220683202925,184.22879727483888,-0.001166397188233491,0.9937389396213173,0.016518025302641348,0.27939135732938875,0.1340973494291039 +3724_NT_T1,14480,14816,2,135,0.009111771058315335,106.67158945327402,0.007199756307591389,107.35530332635601,0.4913541354696226,3325.3634863329744,0.49373078520320923,0.006219142931050459,0.8107593372280942 +3726_NT_T1,747,1041,7,252,0.2420749279538905,320.32325345251036,0.3077072559582232,769.5576551383399,0.0647544098635241,35.814169643571994,0.06281677812183356,0.21019387506063708,0.22204179471322388 +3726_NT_T2,813,1485,2,56,0.03771043771043771,137.61742271132735,0.09267166512547297,342.8049439195663,0.00038347238614448427,1.9536089525427087,0.005127400303616948,0.04127830126763985,0.5909999124026126 +3730_NT_T1,1192,1873,4,96,0.05125467164975974,156.30069146538872,0.08344938145509276,354.777648277818,0.24490431943570973,298.85670611507044,0.24778814206934818,0.04372314663987591,0.7675957558115907 +3730_NT_T2,638,993,5,214,0.21550855991943604,201.01997644773692,0.20243703569762025,393.8308159160509,0.1845497985628088,100.25233241120321,0.26112295799571106,0.18046041594534612,0.7681016504019783 diff --git a/analysis/kptracer/results/tumor_benchmarks_anchored_delta0.1.csv b/analysis/kptracer/results/tumor_benchmarks_anchored_delta0.1.csv new file mode 100644 index 000000000..5b64d5c8c --- /dev/null +++ b/analysis/kptracer/results/tumor_benchmarks_anchored_delta0.1.csv @@ -0,0 +1,22 @@ +tumor,n_cells,n_edges,n_clusters,fitch_changes,fitch_plasticity,thmm_changes,thmm_plasticity,thmm_jumps,path_autocorr_leiden,path_autocorr_z_leiden,path_autocorr_thmm,published_mean_scPlasticity,cell_spearman_vs_published +3430_NT_T2,502,641,4,48,0.0748829953198128,100.00485447872958,0.15601381353935972,725.9332678239443,0.008236228719144442,6.039748163390079,0.0025243570215201247,0.07210215414957862,-0.12058892880548687 +3432_NT_T1,219,355,6,86,0.24225352112676057,73.41415768679869,0.20680044418816532,473.51587420428643,0.10966428215135356,24.65355406744032,0.19281271256521615,0.23482823540233014,0.7929450475427017 +3432_NT_T2,196,294,6,71,0.24149659863945577,50.69947538186237,0.1724471951764026,143.23678541610963,0.11417602954893143,23.783519419233873,0.23132050815111585,0.24251616347783578,0.9323618981888258 +3433_NT_T2,327,417,6,146,0.3501199040767386,136.0050658447355,0.326151237037735,368.2164613882326,0.030279669808902003,9.94760141810501,0.039271804729031023,0.279334560530666,0.418289644102344 +3434_NT_T2,1103,1562,5,366,0.23431498079385404,434.3049353259212,0.2780441327310635,2215.1098805799847,0.007061926992934453,15.101117748645624,0.01459137958450293,0.18559919817458384,0.3911571798926741 +3434_NT_T3,2220,2688,3,695,0.2585565476190476,806.8965930180872,0.3001847444263717,2417.7043728416597,0.0005346996401901756,3.247276721845305,0.0006748111350045944,0.18281316496814853,0.26073897774489246 +3435_NT_T1,1069,1829,4,212,0.11591033351558229,149.16182425249082,0.08155375847593811,991.7537773730237,0.07149819588018567,114.64692541084038,0.11753121223643759,0.11283396391171881,0.2659717088289622 +3435_NT_T3,316,578,4,79,0.13667820069204153,93.99254144323395,0.1626168537080172,783.129317083562,0.013744786038700916,7.44761668431199,0.02233453730581675,0.13783076096852143,0.6518949752223084 +3435_NT_T4,272,402,3,31,0.07711442786069651,51.583143487065115,0.12831627733100776,296.0744961946219,0.0027370681606747505,2.5350750977635665,0.05968584385034842,0.07552029526510093,0.8448601780959171 +3435_NT_T6,168,283,4,29,0.10247349823321555,27.795224614320915,0.09821634139336012,148.4440558645234,0.009869623435229015,3.026562142481424,0.11339382843811195,0.09658270906188489,0.7398595248687123 +3513_NT_T2,2053,2693,4,329,0.12216858522094319,541.5104559687882,0.20108074859591094,3931.9599681350805,0.013945027872155841,29.52369263242612,0.015900433322395574,0.09632015926093254,0.4722603692625214 +3513_NT_T3,1435,1782,5,540,0.30303030303030304,463.01128622930213,0.2598267599491033,2372.1927136399127,0.04039735001696451,37.04850556282406,0.052274392991705526,0.28500866882729725,0.06485027458048757 +3513_NT_T4,594,816,5,271,0.3321078431372549,318.748582221331,0.39062326252614094,2247.5324318905627,0.03097684660967981,16.717879854265348,0.04684149908545151,0.2856110572553209,0.0016505561799102129 +3513_NT_T5,342,518,5,150,0.28957528957528955,169.28939986452713,0.3268135132519829,979.5120907673257,0.03260614721173709,15.953804269700129,0.059265043417931956,0.26264714581399473,0.5779169259240667 +3520_NT_T1,148,281,4,30,0.10676156583629894,39.137458871424876,0.13927921306556895,280.8661793978536,0.03705425172983887,7.110870964981587,0.07850584499840561,0.11786291073108833,0.7777012707337199 +3703_NT_T2,105,187,6,52,0.27807486631016043,78.35709238548633,0.4190218844143654,713.136984375272,-0.001166397188233491,0.9937389396213173,0.016416813119940237,0.27939135732938875,0.1935752380037697 +3724_NT_T1,14480,14816,2,135,0.009111771058315335,104.31370976324011,0.0070406121600459035,105.68455150857987,0.4913541354696226,3325.3634863329744,0.494852453061536,0.006219142931050459,0.812636196589014 +3726_NT_T1,747,1041,7,252,0.2420749279538905,332.10776654766244,0.31902763357124153,2963.5009888015256,0.0647544098635241,35.814169643571994,0.06254373855883331,0.21019387506063708,0.18349116029891122 +3726_NT_T2,813,1485,2,56,0.03771043771043771,101.44289851249826,0.0683117161700325,866.2299874447854,0.00038347238614448427,1.9536089525427087,0.011189006449230215,0.04127830126763985,0.6615551446851986 +3730_NT_T1,1192,1873,4,96,0.05125467164975974,128.95182906608025,0.06884774643143633,1165.2685322059847,0.24490431943570973,298.85670611507044,0.24776943635151819,0.04372314663987591,-0.005594816153303626 +3730_NT_T2,638,993,5,214,0.21550855991943604,171.13265339104439,0.17233902657708397,1360.61316784147,0.1845497985628088,100.25233241120321,0.2944082899022901,0.18046041594534612,0.7667362491176054 diff --git a/analysis/kptracer/results/tumor_benchmarks_anchored_delta2.csv b/analysis/kptracer/results/tumor_benchmarks_anchored_delta2.csv new file mode 100644 index 000000000..bd8352be2 --- /dev/null +++ b/analysis/kptracer/results/tumor_benchmarks_anchored_delta2.csv @@ -0,0 +1,22 @@ +tumor,n_cells,n_edges,n_clusters,fitch_changes,fitch_plasticity,thmm_changes,thmm_plasticity,thmm_jumps,path_autocorr_leiden,path_autocorr_z_leiden,path_autocorr_thmm,published_mean_scPlasticity,cell_spearman_vs_published +3430_NT_T2,502,641,4,48,0.0748829953198128,71.0936968659908,0.11091060353508704,84.93244769602688,0.008236228719144442,6.039748163390079,0.0025243570215201247,0.07210215414957862,0.1826763527916126 +3432_NT_T1,219,355,6,86,0.24225352112676057,64.39764741036431,0.18140182369116709,85.07235552506583,0.10966428215135356,24.65355406744032,0.17585380776726495,0.23482823540233014,0.7232557776765175 +3432_NT_T2,196,294,6,71,0.24149659863945577,63.66746778977037,0.2165560128903754,77.50409910411845,0.11417602954893143,23.783519419233873,0.15252338114357233,0.24251616347783578,0.8817184021225655 +3433_NT_T2,327,417,6,146,0.3501199040767386,128.22396736071136,0.3074915284429529,150.30483114782885,0.030279669808902003,9.94760141810501,0.043100619952630065,0.279334560530666,0.3403489212153558 +3434_NT_T2,1103,1562,5,366,0.23431498079385404,271.65285879704356,0.17391348194433007,324.49428301488774,0.007061926992934453,15.101117748645624,0.01765329598650655,0.18559919817458384,0.22361876743969053 +3434_NT_T3,2220,2688,3,695,0.2585565476190476,673.707856626217,0.25063536332820574,707.1095804277554,0.0005346996401901756,3.247276721845305,0.00046549531303636063,0.18281316496814853,0.14133813400649822 +3435_NT_T1,1069,1829,4,212,0.11591033351558229,166.37767797573744,0.09096647237601828,233.68597145027968,0.07149819588018567,114.64692541084038,0.11252763702087204,0.11283396391171881,0.45624838455131905 +3435_NT_T3,316,578,4,79,0.13667820069204153,93.45789646920628,0.16169186240347105,130.8200542430476,0.013744786038700916,7.44761668431199,0.01819156711553197,0.13783076096852143,0.7362504445750129 +3435_NT_T4,272,402,3,31,0.07711442786069651,36.22399239788442,0.09010943382558315,53.45051470070558,0.0027370681606747505,2.5350750977635665,0.024098387486467303,0.07552029526510093,0.28388173131317307 +3435_NT_T6,168,283,4,29,0.10247349823321555,27.360398389128665,0.09667985296511895,40.429839382944685,0.009869623435229015,3.026562142481424,0.09508811528699455,0.09658270906188489,0.7559749610864827 +3513_NT_T2,2053,2693,4,329,0.12216858522094319,370.04889394011957,0.13741139767549929,522.7231493981,0.013945027872155841,29.52369263242612,0.015991327632376436,0.09632015926093254,0.2157433211847245 +3513_NT_T3,1435,1782,5,540,0.30303030303030304,447.25614661134694,0.25098549192555947,577.6232602071636,0.04039735001696451,37.04850556282406,0.04992055777109489,0.28500866882729725,-0.026521628276471202 +3513_NT_T4,594,816,5,271,0.3321078431372549,241.98703763560448,0.2965527422004957,325.63541260642546,0.03097684660967981,16.717879854265348,0.041177102422748184,0.2856110572553209,-0.31998262629360635 +3513_NT_T5,342,518,5,150,0.28957528957528955,156.4168502488697,0.30196303136847435,196.11841127777404,0.03260614721173709,15.953804269700129,0.04512493297551282,0.26264714581399473,0.43841426344211676 +3520_NT_T1,148,281,4,30,0.10676156583629894,36.98154279343042,0.13160691385562429,52.04087006149251,0.03705425172983887,7.110870964981587,0.07602920881293122,0.11786291073108833,0.80198152692247 +3703_NT_T2,105,187,6,52,0.27807486631016043,63.22924086105669,0.33812428267944755,83.06474580394868,-0.001166397188233491,0.9937389396213173,0.018150020999175755,0.27939135732938875,-0.24517822852172513 +3724_NT_T1,14480,14816,2,135,0.009111771058315335,107.0708268440506,0.0072267026757593545,107.63246950928293,0.4913541354696226,3325.3634863329744,0.49397769146925324,0.006219142931050459,0.8072101261589841 +3726_NT_T1,747,1041,7,252,0.2420749279538905,282.9163361864323,0.27177361785440185,365.2951834362974,0.0647544098635241,35.814169643571994,0.05983732387131738,0.21019387506063708,0.29930345574928163 +3726_NT_T2,813,1485,2,56,0.03771043771043771,100.52845723496782,0.06769593079795813,168.6063843351313,0.00038347238614448427,1.9536089525427087,0.004244484355541557,0.04127830126763985,0.6905397452427264 +3730_NT_T1,1192,1873,4,96,0.05125467164975974,130.70373634333197,0.06978309468410676,195.88497915383948,0.24490431943570973,298.85670611507044,0.24778814206934818,0.04372314663987591,0.32914799163422087 +3730_NT_T2,638,993,5,214,0.21550855991943604,183.77210841609917,0.1850675814865047,249.7272453146843,0.1845497985628088,100.25233241120321,0.24792096194468466,0.18046041594534612,0.7042691345138008 diff --git a/analysis/kptracer/results/tumor_benchmarks_anchored_freeE.csv b/analysis/kptracer/results/tumor_benchmarks_anchored_freeE.csv new file mode 100644 index 000000000..ac068d797 --- /dev/null +++ b/analysis/kptracer/results/tumor_benchmarks_anchored_freeE.csv @@ -0,0 +1,22 @@ +tumor,n_cells,n_edges,n_clusters,fitch_changes,fitch_plasticity,thmm_changes,thmm_plasticity,thmm_jumps,path_autocorr_leiden,path_autocorr_z_leiden,path_autocorr_thmm,published_mean_scPlasticity,cell_spearman_vs_published +3430_NT_T2,502,641,4,48,0.0748829953198128,31.77141657578204,0.04956539247391894,47.70000314596503,0.008236228719144442,6.039748163390079,0.007964947062443922,0.07210215414957862,-0.09057476494599682 +3432_NT_T1,219,355,6,86,0.24225352112676057,66.05187626713297,0.1860616232876985,106.40763858209525,0.10966428215135356,24.65355406744032,0.19012682383019947,0.23482823540233014,0.7114202414783081 +3432_NT_T2,196,294,6,71,0.24149659863945577,56.24725722707078,0.1913172014526217,80.63262582732895,0.11417602954893143,23.783519419233873,0.157757367787943,0.24251616347783578,0.6560007720022585 +3433_NT_T2,327,417,6,146,0.3501199040767386,92.91760597537213,0.22282399514477727,124.66362230207275,0.030279669808902003,9.94760141810501,0.06899242755059586,0.279334560530666,0.37579514930319224 +3434_NT_T2,1103,1562,5,366,0.23431498079385404,72.91481759583638,0.046680420996054024,115.42315841785098,0.007061926992934453,15.101117748645624,0.03551332225095007,0.18559919817458384,0.01632593951997569 +3434_NT_T3,2220,2688,3,695,0.2585565476190476,898.085297421052,0.3341091136239033,2005.2587168558587,0.0005346996401901756,3.247276721845305,0.0008294420865433689,0.18281316496814853,0.2155972462923535 +3435_NT_T1,1069,1829,4,212,0.11591033351558229,267.82008176009737,0.14642978773105378,349.51517414004724,0.07149819588018567,114.64692541084038,0.06986574036086363,0.11283396391171881,0.3692954973903566 +3435_NT_T3,316,578,4,79,0.13667820069204153,105.41675255919971,0.18238192484290608,151.67631221131765,0.013744786038700916,7.44761668431199,0.029600613179458692,0.13783076096852143,0.5740482356615256 +3435_NT_T4,272,402,3,31,0.07711442786069651,39.70743666154416,0.09877471806354268,58.52808187315373,0.0027370681606747505,2.5350750977635665,0.07425487249776148,0.07552029526510093,-0.25341594591967526 +3435_NT_T6,168,283,4,29,0.10247349823321555,39.129496994847564,0.13826677383338362,51.007431529807846,0.009869623435229015,3.026562142481424,0.09898817680867023,0.09658270906188489,0.5044971989323344 +3513_NT_T2,2053,2693,4,329,0.12216858522094319,505.8187322020904,0.18782723067288912,875.9317390221752,0.013945027872155841,29.52369263242612,0.01494343375067912,0.09632015926093254,0.24160779670128335 +3513_NT_T3,1435,1782,5,540,0.30303030303030304,488.66826303121604,0.2742246144956319,706.5776466805705,0.04039735001696451,37.04850556282406,0.03816916398044888,0.28500866882729725,0.08488671818165383 +3513_NT_T4,594,816,5,271,0.3321078431372549,193.0390807542773,0.23656750092435944,367.9334643949892,0.03097684660967981,16.717879854265348,0.062342519067231206,0.2856110572553209,-0.2491611952873142 +3513_NT_T5,342,518,5,150,0.28957528957528955,113.34729841353396,0.2188171784045057,172.53571110750642,0.03260614721173709,15.953804269700129,0.07579329544143365,0.26264714581399473,0.49870695172715446 +3520_NT_T1,148,281,4,30,0.10676156583629894,51.403190166997504,0.1829295023736566,72.55595831940335,0.03705425172983887,7.110870964981587,0.05051541875023136,0.11786291073108833,0.8110856973964158 +3703_NT_T2,105,187,6,52,0.27807486631016043,55.107552245061925,0.29469279275434185,79.85997467496334,-0.001166397188233491,0.9937389396213173,0.027414942122473984,0.27939135732938875,0.06908142279016619 +3724_NT_T1,14480,14816,2,135,0.009111771058315335,4525.260020928178,0.305430616963295,5401.081055521876,0.4913541354696226,3325.3634863329744,0.27227218061445724,0.006219142931050459,0.7660720807009403 +3726_NT_T1,747,1041,7,252,0.2420749279538905,97.4413710086962,0.09360362248673985,202.0826023680341,0.0647544098635241,35.814169643571994,0.1485422329816427,0.21019387506063708,0.433398859547515 +3726_NT_T2,813,1485,2,56,0.03771043771043771,152.17827500252784,0.10247695286365512,251.14234308224817,0.00038347238614448427,1.9536089525427087,0.016452868896528465,0.04127830126763985,0.49744178071626266 +3730_NT_T1,1192,1873,4,96,0.05125467164975974,198.57191654388149,0.10601810813875145,323.9140147286077,0.24490431943570973,298.85670611507044,0.20276607445774833,0.04372314663987591,0.3749250024456015 +3730_NT_T2,638,993,5,214,0.21550855991943604,177.59127982456425,0.17884318209925906,324.1194532362041,0.1845497985628088,100.25233241120321,0.23313242227729386,0.18046041594534612,0.7004254269316111 diff --git a/analysis/kptracer/results/tumor_benchmarks_anchored_no3724.csv b/analysis/kptracer/results/tumor_benchmarks_anchored_no3724.csv new file mode 100644 index 000000000..0562023b1 --- /dev/null +++ b/analysis/kptracer/results/tumor_benchmarks_anchored_no3724.csv @@ -0,0 +1,21 @@ +tumor,n_cells,n_edges,n_clusters,fitch_changes,fitch_plasticity,thmm_changes,thmm_plasticity,thmm_jumps,path_autocorr_leiden,path_autocorr_z_leiden,path_autocorr_thmm,published_mean_scPlasticity,cell_spearman_vs_published +3430_NT_T2,502,641,4,48,0.0748829953198128,87.72794963148996,0.13686107586815907,218.089746062105,0.008236228719144442,6.039748163390079,0.002715937585085175,0.07210215414957862,-0.12025025258993803 +3432_NT_T1,219,355,6,86,0.24225352112676057,70.5384030493404,0.19869972689955043,126.3481683618213,0.10966428215135356,24.65355406744032,0.19162725283533677,0.23482823540233014,0.7231816695633647 +3432_NT_T2,196,294,6,71,0.24149659863945577,66.69468498650414,0.22685267002212295,91.40420175082502,0.11417602954893143,23.783519419233873,0.16291885629948746,0.24251616347783578,0.8840277354807323 +3433_NT_T2,327,417,6,146,0.3501199040767386,135.49656852020206,0.3249318189932903,175.24639549222633,0.030279669808902003,9.94760141810501,0.0350878983559358,0.279334560530666,0.43174783203066674 +3434_NT_T2,1103,1562,5,366,0.23431498079385404,300.3739860498332,0.19230088735584713,473.31916281344655,0.007061926992934453,15.101117748645624,0.018185178924751345,0.18559919817458384,0.06377728032434475 +3434_NT_T3,2220,2688,3,695,0.2585565476190476,666.1336411912079,0.24781757484791958,697.7557341433792,0.0005346996401901756,3.247276721845305,0.0007043381489655953,0.18281316496814853,0.09372112996824439 +3435_NT_T1,1069,1829,4,212,0.11591033351558229,201.31548455249242,0.1100686082845776,393.1754937032267,0.07149819588018567,114.64692541084038,0.11096387574573752,0.11283396391171881,0.42213607369208606 +3435_NT_T3,316,578,4,79,0.13667820069204153,106.54641182895139,0.18433635264524462,217.38438264651847,0.013744786038700916,7.44761668431199,0.020067992147329714,0.13783076096852143,0.684176259842451 +3435_NT_T4,272,402,3,31,0.07711442786069651,46.92638937652718,0.11673231188190841,90.1437742408086,0.0027370681606747505,2.5350750977635665,0.04106248327168731,0.07552029526510093,0.6542451912793197 +3435_NT_T6,168,283,4,29,0.10247349823321555,36.02039944946102,0.1272805634256573,66.96466494251618,0.009869623435229015,3.026562142481424,0.10203821372237794,0.09658270906188489,0.8262806400619819 +3513_NT_T2,2053,2693,4,329,0.12216858522094319,437.6421197900952,0.16251099880805614,893.3359479943063,0.013945027872155841,29.52369263242612,0.016147502520402614,0.09632015926093254,0.3009910544439415 +3513_NT_T3,1435,1782,5,540,0.30303030303030304,467.1113860075928,0.26212760157552906,771.6231341173037,0.04039735001696451,37.04850556282406,0.04881139390417036,0.28500866882729725,-0.010842284504797088 +3513_NT_T4,594,816,5,271,0.3321078431372549,300.93901914510894,0.36879781757979035,629.4557480436051,0.03097684660967981,16.717879854265348,0.04214716332976394,0.2856110572553209,-0.2828179780480706 +3513_NT_T5,342,518,5,150,0.28957528957528955,169.2786697706316,0.3267927987850031,278.0615751941839,0.03260614721173709,15.953804269700129,0.05253377823218697,0.26264714581399473,0.45309833572820163 +3520_NT_T1,148,281,4,30,0.10676156583629894,46.45378644466795,0.1653159659952596,92.32950627535725,0.03705425172983887,7.110870964981587,0.07665930417022658,0.11786291073108833,0.805346997016336 +3703_NT_T2,105,187,6,52,0.27807486631016043,77.55893810915363,0.4147536797280943,200.62822915145125,-0.001166397188233491,0.9937389396213173,0.016518025302641348,0.27939135732938875,0.1569644314555676 +3726_NT_T1,747,1041,7,252,0.2420749279538905,321.53680224097127,0.3088730088770137,813.837588642277,0.0647544098635241,45.513665685108954,0.06268588997735078,0.21019387506063708,0.2219947853511853 +3726_NT_T2,813,1485,2,56,0.03771043771043771,136.98088561438195,0.09224302061574542,338.9031033423035,0.00038347238614448427,2.6990626114468506,0.005127400303616948,0.04127830126763985,0.5933377351547607 +3730_NT_T1,1192,1873,4,96,0.05125467164975974,155.67115427746236,0.08311326976906694,351.8859183234074,0.24490431943570973,278.7890803670364,0.24778814206934818,0.04372314663987591,0.766410043461216 +3730_NT_T2,638,993,5,214,0.21550855991943604,197.6644341955593,0.1990578390690426,384.8141851973388,0.1845497985628088,98.05104547810004,0.2652792836857237,0.18046041594534612,0.7543897845430149 diff --git a/analysis/kptracer/results/tumor_benchmarks_anchored_unit.csv b/analysis/kptracer/results/tumor_benchmarks_anchored_unit.csv new file mode 100644 index 000000000..d62f43101 --- /dev/null +++ b/analysis/kptracer/results/tumor_benchmarks_anchored_unit.csv @@ -0,0 +1,22 @@ +tumor,n_cells,n_edges,n_clusters,fitch_changes,fitch_plasticity,thmm_changes,thmm_plasticity,thmm_jumps,path_autocorr_leiden,path_autocorr_z_leiden,path_autocorr_thmm,published_mean_scPlasticity,cell_spearman_vs_published +3430_NT_T2,502,641,4,48,0.0748829953198128,71.62429199553176,0.11173836504763146,84.12284966243382,0.008236228719144442,6.039748163390079,0.0025243570215201247,0.07210215414957862,0.2077428596377312 +3432_NT_T1,219,355,6,86,0.24225352112676057,62.304568672435124,0.17550582724629613,76.13662185378355,0.10966428215135356,24.65355406744032,0.1747985259231502,0.23482823540233014,0.7661879015771331 +3432_NT_T2,196,294,6,71,0.24149659863945577,63.12506198720848,0.21471109519458667,76.03140405323437,0.11417602954893143,23.783519419233873,0.15450765443307932,0.24251616347783578,0.8766913674521917 +3433_NT_T2,327,417,6,146,0.3501199040767386,128.7344693896555,0.30871575393202755,155.67586423657798,0.030279669808902003,9.94760141810501,0.045585153228551833,0.279334560530666,0.31590758015418646 +3434_NT_T2,1103,1562,5,366,0.23431498079385404,271.79653887513075,0.17400546662940508,301.9695975446543,0.007061926992934453,15.101117748645624,0.014129230662320033,0.18559919817458384,0.2677447958941002 +3434_NT_T3,2220,2688,3,695,0.2585565476190476,678.6300319160783,0.2524665297306839,752.5439221474743,0.0005346996401901756,3.247276721845305,0.0004861221249179165,0.18281316496814853,0.15310874378401584 +3435_NT_T1,1069,1829,4,212,0.11591033351558229,141.38833992171638,0.07730363035632389,171.10703903972868,0.07149819588018567,114.64692541084038,0.11364074412432662,0.11283396391171881,0.37498528691173205 +3435_NT_T3,316,578,4,79,0.13667820069204153,79.05254478856835,0.1367691086307411,90.92513478909837,0.013744786038700916,7.44761668431199,0.017864015970446782,0.13783076096852143,0.6652269559118956 +3435_NT_T4,272,402,3,31,0.07711442786069651,34.473860253057865,0.08575587127626334,47.63553834884152,0.0027370681606747505,2.5350750977635665,0.014661925090321415,0.07552029526510093,0.4447046528818706 +3435_NT_T6,168,283,4,29,0.10247349823321555,24.147032520438586,0.08532520325243316,32.02320833024593,0.009869623435229015,3.026562142481424,0.08901451186374322,0.09658270906188489,0.7037326963635597 +3513_NT_T2,2053,2693,4,329,0.12216858522094319,344.89766044498714,0.12807191253063022,456.25042309109267,0.013945027872155841,29.52369263242612,0.015563405531599083,0.09632015926093254,0.23734041573475437 +3513_NT_T3,1435,1782,5,540,0.30303030303030304,427.1599756497911,0.23970817937698713,520.4821007684662,0.04039735001696451,37.04850556282406,0.05040735461175522,0.28500866882729725,0.4696981176189632 +3513_NT_T4,594,816,5,271,0.3321078431372549,232.2806055854816,0.28465760488416864,286.85264131478647,0.03097684660967981,16.717879854265348,0.04145273596808152,0.2856110572553209,-0.5579821056301234 +3513_NT_T5,342,518,5,150,0.28957528957528955,151.20355577570666,0.29189875632375806,177.02261735895388,0.03260614721173709,15.953804269700129,0.03639356098200256,0.26264714581399473,0.5215264876534461 +3520_NT_T1,148,281,4,30,0.10676156583629894,29.61661454445264,0.10539720478452898,35.133141746679605,0.03705425172983887,7.110870964981587,0.07602920881293122,0.11786291073108833,0.6097276594151153 +3703_NT_T2,105,187,6,52,0.27807486631016043,57.160962560277575,0.3056735965790245,66.62740872401679,-0.001166397188233491,0.9937389396213173,0.021010679236372058,0.27939135732938875,-0.1839503626011237 +3724_NT_T1,14480,14816,2,135,0.009111771058315335,107.07721685145012,0.007227133966755542,107.67117125085822,0.4913541354696226,3325.3634863329744,0.49397769146925324,0.006219142931050459,0.8028333456655029 +3726_NT_T1,747,1041,7,252,0.2420749279538905,262.5096173052199,0.2521706218109701,294.7994368482059,0.0647544098635241,35.814169643571994,0.060181588182871684,0.21019387506063708,0.11283738113402177 +3726_NT_T2,813,1485,2,56,0.03771043771043771,74.03253441307862,0.04985355852732567,94.91916873625249,0.00038347238614448427,1.9536089525427087,0.0016277452090439205,0.04127830126763985,0.5491676765105413 +3730_NT_T1,1192,1873,4,96,0.05125467164975974,108.45562860331964,0.05790476700657749,129.68294196488105,0.24490431943570973,298.85670611507044,0.2483193329361814,0.04372314663987591,0.07938815507840477 +3730_NT_T2,638,993,5,214,0.21550855991943604,168.07550015287435,0.16926032240974256,194.7549388914187,0.1845497985628088,100.25233241120321,0.24164436500605618,0.18046041594534612,0.6847372319474109 diff --git a/analysis/kptracer/results/tumor_benchmarks_unsup.csv b/analysis/kptracer/results/tumor_benchmarks_unsup.csv new file mode 100644 index 000000000..1557b8ca7 --- /dev/null +++ b/analysis/kptracer/results/tumor_benchmarks_unsup.csv @@ -0,0 +1,22 @@ +tumor,n_cells,n_edges,n_clusters,fitch_changes,fitch_plasticity,thmm_changes,thmm_plasticity,thmm_jumps,path_autocorr_leiden,path_autocorr_z_leiden,path_autocorr_thmm,published_mean_scPlasticity,cell_spearman_vs_published +3430_NT_T2,502,641,4,48,0.0748829953198128,9.41155544268318,0.014682613795137568,10.484092642327521,0.008236228719144442,6.039748163390079,0.020742049446356197,0.07210215414957862,0.016799328710653146 +3432_NT_T1,219,355,6,86,0.24225352112676057,32.11373206416745,0.09046121708216183,45.84688171860035,0.10966428215135356,24.65355406744032,0.2363448857600423,0.23482823540233014,0.6655002666664516 +3432_NT_T2,196,294,6,71,0.24149659863945577,25.246337532356108,0.08587189636855819,30.25757490011859,0.11417602954893143,23.783519419233873,0.2802200265165788,0.24251616347783578,0.7067230906304895 +3433_NT_T2,327,417,6,146,0.3501199040767386,81.54468615455087,0.19555080612602127,89.09624019241716,0.030279669808902003,9.94760141810501,0.06852023729336074,0.279334560530666,0.2835088295813239 +3434_NT_T2,1103,1562,5,366,0.23431498079385404,44.88511243640896,0.02873566737286105,113.58385794770443,0.007061926992934453,15.101117748645624,0.04457190995128954,0.18559919817458384,0.023872038201768208 +3434_NT_T3,2220,2688,3,695,0.2585565476190476,654.6488729054943,0.2435449675987702,702.7151435712757,0.0005346996401901756,3.247276721845305,0.0012155317068867344,0.18281316496814853,0.08396864109335933 +3435_NT_T1,1069,1829,4,212,0.11591033351558229,142.80643017511048,0.07807896674418288,188.23992612753239,0.07149819588018567,114.64692541084038,0.10566780797925131,0.11283396391171881,0.29155208898688595 +3435_NT_T3,316,578,4,79,0.13667820069204153,74.3676315070295,0.12866372233050086,104.01381000679477,0.013744786038700916,7.44761668431199,0.02522156408439792,0.13783076096852143,0.6695435210120227 +3435_NT_T4,272,402,3,31,0.07711442786069651,39.8633597020495,0.09916258632350622,48.482545440646646,0.0027370681606747505,2.5350750977635665,0.1267826888739173,0.07552029526510093,0.39691586909267756 +3435_NT_T6,168,283,4,29,0.10247349823321555,24.24490013424034,0.08567102520932982,31.547693973462376,0.009869623435229015,3.026562142481424,0.09707775730245959,0.09658270906188489,0.7704246798943736 +3513_NT_T2,2053,2693,4,329,0.12216858522094319,393.9677533895153,0.1462932615631323,443.2364366672127,0.013945027872155841,29.52369263242612,0.01569227960335903,0.09632015926093254,0.19274623088394227 +3513_NT_T3,1435,1782,5,540,0.30303030303030304,548.777550255797,0.30795597657452134,618.9978230150728,0.04039735001696451,37.04850556282406,0.04589948322152595,0.28500866882729725,0.30901846767773583 +3513_NT_T4,594,816,5,271,0.3321078431372549,109.8933184724267,0.1346731844024837,146.9936815661968,0.03097684660967981,16.717879854265348,0.13201714668550601,0.2856110572553209,-0.07004977628298092 +3513_NT_T5,342,518,5,150,0.28957528957528955,81.28149712070312,0.15691408710560448,106.66496887436783,0.03260614721173709,15.953804269700129,0.08115684409751472,0.26264714581399473,0.26979640500484847 +3520_NT_T1,148,281,4,30,0.10676156583629894,20.94727824788644,0.0745454741917667,31.24635351418154,0.03705425172983887,7.110870964981587,0.08867770694732514,0.11786291073108833,0.7447740889077592 +3703_NT_T2,105,187,6,52,0.27807486631016043,36.7601492266447,0.19657833811039946,45.31069811589033,-0.001166397188233491,0.9937389396213173,0.05967330652410016,0.27939135732938875,-0.2661619507825935 +3724_NT_T1,14480,14816,2,135,0.009111771058315335,6437.252557894165,0.43447978927471415,8411.189351896754,0.4913541354696226,3325.3634863329744,0.12020185559638666,0.006219142931050459,-0.7272353161523533 +3726_NT_T1,747,1041,7,252,0.2420749279538905,119.02400667794956,0.11433622159265087,153.87993652690767,0.0647544098635241,35.814169643571994,0.1413609947288723,0.21019387506063708,0.4007811023767407 +3726_NT_T2,813,1485,2,56,0.03771043771043771,87.47554942619892,0.05890609388969625,136.3353484606296,0.00038347238614448427,1.9536089525427087,0.007931683920473755,0.04127830126763985,0.585713518478525 +3730_NT_T1,1192,1873,4,96,0.05125467164975974,127.42453611729563,0.06803232040432228,184.0702975999858,0.24490431943570973,298.85670611507044,0.23587427632795463,0.04372314663987591,-0.035439829887657634 +3730_NT_T2,638,993,5,214,0.21550855991943604,93.97446665497633,0.09463692513089257,149.36703739104578,0.1845497985628088,100.25233241120321,0.2891579157700678,0.18046041594534612,0.6362274436118486 diff --git a/analysis/kptracer/scalability.py b/analysis/kptracer/scalability.py new file mode 100644 index 000000000..85b55b9c5 --- /dev/null +++ b/analysis/kptracer/scalability.py @@ -0,0 +1,55 @@ +"""Runtime of the E-step and of a full fit against tree size. Writes ``results/scalability.csv``.""" + +import os + +os.environ.setdefault("OMP_NUM_THREADS", "1") +os.environ.setdefault("OPENBLAS_NUM_THREADS", "1") + +import time +from pathlib import Path + +import numpy as np +import pandas as pd +from simulation_study import PI_TRUE, Q_TRUE, D, K + +from lineage.phylo_sim import simulate_dataset +from lineage.phyloHMM import PhyloHMM +from lineage.states.StateDistributionGaussian import StateDistribution as Gaussian +from lineage.tree_io import to_lineage + +OUT = Path(__file__).parent / "results" + +if __name__ == "__main__": + rows = [] + rng = np.random.default_rng(0) + means = rng.normal(0, 1.0, (K, D)) + for n_leaves in [100, 300, 1000, 3000, 10000, 30000]: + data = simulate_dataset(1, n_leaves, Q_TRUE, PI_TRUE, means, 1.0, rng=rng) + d = data[0] + X = [to_lineage(d["tree"], d["tree"].leaf_names, d["obs"][d["tree"].is_leaf], [Gaussian(dim=D)])] + m = PhyloHMM(X, K, rng=0) + m.init_emissions() + m.e_step() # warm up + t0 = time.perf_counter() + reps = 3 + for _ in range(reps): + post = m.e_step() + t_e = (time.perf_counter() - t0) / reps + t0 = time.perf_counter() + m.m_step(post) + t_m = time.perf_counter() - t0 + t0 = time.perf_counter() + m2 = PhyloHMM(X, K, rng=0).fit(tol=1e-5) + t_fit = time.perf_counter() - t0 + rows.append( + { + "n_leaves": n_leaves, + "n_nodes": len(X[0]), + "estep_s": t_e, + "mstep_s": t_m, + "fit_s": t_fit, + "iters": len(m2.LL_trace), + } + ) + print(rows[-1], flush=True) + pd.DataFrame(rows).to_csv(OUT / "scalability.csv", index=False) diff --git a/analysis/kptracer/simulation_study.py b/analysis/kptracer/simulation_study.py new file mode 100644 index 000000000..d4beadfdd --- /dev/null +++ b/analysis/kptracer/simulation_study.py @@ -0,0 +1,160 @@ +"""Simulation study: recovery of Q, emissions, and ancestral states as leaf sampling drops. + +Run with ``uv run python analysis/kptracer/simulation_study.py``. Writes ``results/simulation.csv``. +""" + +import os + +os.environ.setdefault("OMP_NUM_THREADS", "1") +os.environ.setdefault("OPENBLAS_NUM_THREADS", "1") + +import itertools +from concurrent.futures import ProcessPoolExecutor +from pathlib import Path + +import numpy as np +import pandas as pd +from scipy.optimize import linear_sum_assignment +from sklearn.cluster import KMeans + +from lineage import ctmc +from lineage.phylo_sim import simulate_dataset +from lineage.phylo_stats import small_parsimony +from lineage.phyloHMM import fit_best +from lineage.states.StateDistributionGaussian import StateDistribution as Gaussian +from lineage.tree_io import to_lineage + +OUT = Path(__file__).parent / "results" + +# A progression-like chain: 0 -> 1 -> 2 -> 3 with slow reversions, loosely modeled on the +# AT2-like -> gastric-like -> pre-EMT -> mesenchymal paths reported for KP-Tracer tumors. +RATES = np.array( + [ + [0.000, 0.150, 0.020, 0.005], + [0.050, 0.000, 0.100, 0.005], + [0.005, 0.020, 0.000, 0.080], + [0.005, 0.005, 0.010, 0.000], + ] +) +Q_TRUE = ctmc.rates_to_Q(RATES) +PI_TRUE = np.array([0.85, 0.10, 0.04, 0.01]) +K, D = 4, 10 +N_TREES, N_LEAVES = 20, 500 + + +def match_states(est_means: np.ndarray, true_means: np.ndarray) -> np.ndarray: + """perm[true state] = estimated state, by minimum total distance between means.""" + C = np.linalg.norm(est_means[:, None, :] - true_means[None, :, :], axis=2) + r, c = linear_sum_assignment(C) + perm = np.empty(K, dtype=int) + perm[c] = r + return perm + + +def true_edge_changes(data) -> tuple[np.ndarray, np.ndarray]: + """Per-tree number of edges of the sampled tree whose endpoint states differ, and edge counts.""" + changes, edges = [], [] + for d in data: + par = d["tree"].parents[1:] + s = d["states"] + changes.append(int(np.sum(s[par] != s[1:]))) + edges.append(len(par)) + return np.array(changes), np.array(edges) + + +def run_one(args): + sep, frac, seed = args + rng = np.random.default_rng(seed) + means = rng.normal(0, sep, (K, D)) + data = simulate_dataset(N_TREES, N_LEAVES, Q_TRUE, PI_TRUE, means, 1.0, sample_frac=frac, rng=rng) + X = [to_lineage(d["tree"], d["tree"].leaf_names, d["obs"][d["tree"].is_leaf], [Gaussian(dim=D)]) for d in data] + n_leaves = int(sum(d["tree"].is_leaf.sum() for d in data)) + tc, n_edges = true_edge_changes(data) + rows = [] + + for mode in ("ctmc", "discrete"): + if mode == "discrete": + for x in X: + x.branch_lengths = None + m = fit_best(X, K, n_init=3, rng=seed, mode=mode, tol=1e-5) + perm = match_states(np.array([e.mean for e in m.E]), means) + leaf_acc, anc_acc = [], [] + for p, d in zip(m.posteriors, data, strict=True): + est = np.argmax(p.gamma[:, perm], axis=1) + leaf = d["tree"].is_leaf + leaf_acc.append(np.mean(est[leaf] == d["states"][leaf])) + anc_acc.append(np.mean(est[~leaf] == d["states"][~leaf])) + summ = m.switch_summary() + est_changes = np.array([s["expected_changes"] for s in summ]) + row = { + "method": f"tHMM-{mode}", + "sep": sep, + "frac": frac, + "seed": seed, + "n_leaves": n_leaves, + "leaf_acc": np.mean(leaf_acc), + "anc_acc": np.mean(anc_acc), + "mean_rmse": float(np.sqrt(np.mean((np.array([e.mean for e in m.E])[perm] - means) ** 2))), + "changes_ratio": est_changes.sum() / tc.sum(), + "plasticity_corr": np.corrcoef(est_changes / n_edges, tc / n_edges)[0, 1], + "iters": len(m.LL_trace), + } + if mode == "ctmc": + Qhat = m.Q[np.ix_(perm, perm)] + off = ~np.eye(K, dtype=bool) + row["Q_rel_err"] = float(np.linalg.norm(Qhat - Q_TRUE) / np.linalg.norm(Q_TRUE)) + row["exit_rate_rel_err"] = float(np.mean(np.abs(np.diag(Qhat) / np.diag(Q_TRUE) - 1))) + big = RATES > 0.04 + row["big_rate_rel_err"] = float(np.mean(np.abs(Qhat[big] / RATES[big] - 1))) + row["small_rate_abs_err"] = float(np.mean(np.abs(Qhat[off & ~big] - RATES[off & ~big]))) + row["pi_tv"] = 0.5 * float(np.abs(m.pi[perm] - PI_TRUE).sum()) + for i, j in zip(*np.nonzero(off), strict=True): + row[f"q{i}{j}"] = Qhat[i, j] + rows.append(row) + + # Cluster-then-parsimony baseline, with k-means labels and with the true leaf labels (oracle) + obs = np.vstack([d["obs"][d["tree"].is_leaf] for d in data]) + km = KMeans(K, n_init=5, random_state=seed).fit(obs) + perm = match_states(km.cluster_centers_, means) + inv = np.argsort(perm) # estimated label -> true label + for label_source in ("kmeans", "oracle"): + leaf_acc, anc_acc, changes = [], [], [] + if label_source == "kmeans": + lab_iter = iter(inv[km.labels_]) + for d in data: + leaf = d["tree"].is_leaf + lab = np.full(len(leaf), -1) + if label_source == "kmeans": + lab[leaf] = [next(lab_iter) for _ in range(leaf.sum())] + else: + lab[leaf] = d["states"][leaf] + score, labels, _ = small_parsimony(d["tree"], lab, K) + leaf_acc.append(np.mean(labels[leaf] == d["states"][leaf])) + anc_acc.append(np.mean(labels[~leaf] == d["states"][~leaf])) + changes.append(score) + changes = np.array(changes) + rows.append( + { + "method": f"parsimony-{label_source}", + "sep": sep, + "frac": frac, + "seed": seed, + "n_leaves": n_leaves, + "leaf_acc": np.mean(leaf_acc), + "anc_acc": np.mean(anc_acc), + "changes_ratio": changes.sum() / tc.sum(), + "plasticity_corr": np.corrcoef(changes / n_edges, tc / n_edges)[0, 1], + } + ) + return rows + + +if __name__ == "__main__": + OUT.mkdir(exist_ok=True) + grid = list(itertools.product([1.0, 0.5], [1.0, 0.5, 0.25, 0.1], range(5))) + with ProcessPoolExecutor(40) as ex: + rows = [r for res in ex.map(run_one, grid) for r in res] + df = pd.DataFrame(rows) + df.to_csv(OUT / "simulation.csv", index=False) + cols = ["leaf_acc", "anc_acc", "changes_ratio", "plasticity_corr", "Q_rel_err", "big_rate_rel_err", "pi_tv"] + print(df.groupby(["sep", "frac", "method"])[cols].mean().round(3).to_string()) diff --git a/lineage/HMM/E_step.py b/lineage/HMM/E_step.py index ecd44036d..457cfeaaa 100644 --- a/lineage/HMM/E_step.py +++ b/lineage/HMM/E_step.py @@ -3,6 +3,17 @@ from scipy.sparse import csr_array +def edge_T(T: np.ndarray, children: np.ndarray) -> np.ndarray: + """Transition matrices for the edges into ``children``, shape (len(children), K, K). + + ``T`` is either one K by K matrix shared by every edge, or an N by K by K array whose entry + ``T[c]`` is the transition matrix along the edge from the parent of cell c into c. + """ + if T.ndim == 2: + return np.broadcast_to(T, (len(children), *T.shape)) + return T[children] + + def get_MSD( tree: csr_array, pi: npt.NDArray[np.float64], @@ -27,7 +38,7 @@ def get_MSD( :param tree: CSR array representing the lineage tree adjacency :param pi: Initial probabilities vector - :param T: State transitions matrix + :param T: State transitions matrix, shared (K by K) or per edge (N by K by K) :return: The marginal state distribution """ m = np.zeros((tree.shape[0], pi.size)) @@ -37,9 +48,11 @@ def get_MSD( for pIDX in range(tree.shape[0]): start, end = tree.indptr[pIDX], tree.indptr[pIDX + 1] if start < end: - p_trans = m[pIDX, :] @ T - for cIDX in tree.indices[start:end]: - m[cIDX, :] = p_trans + children = tree.indices[start:end] + if T.ndim == 2: + m[children, :] = m[pIDX, :] @ T + else: + m[children, :] = np.einsum("k,ckl->cl", m[pIDX, :], T[children]) # Assert all ~= 1.0 assert np.linalg.norm(np.sum(m, axis=1) - 1.0) < 1e-9 @@ -63,24 +76,42 @@ def get_beta_and_NF( :param leaves_idx: array of indices corresponding to leaf cells :param tree: CSR array representing the lineage tree adjacency - :param T: Transition probability matrix + :param T: Transition probability matrix, shared (K by K) or per edge (N by K by K) :param MSD: The marginal state distribution P(z_n = k) :param EL: The emissions likelihood :return: normalizing factor. The marginal observation distribution P(x_n = x) :return: beta values. The conditional probability of states, given observations of the sub-tree rooted in cell_n """ + logNF, beta = get_beta_and_logNF(leaves_idx, tree, T, MSD, EL) + return np.exp(logNF), beta + + +def get_beta_and_logNF( + leaves_idx: np.ndarray, + tree: csr_array, + T: np.ndarray, + MSD: np.ndarray, + EL: np.ndarray, +) -> tuple[np.ndarray, np.ndarray]: + """ + Same as :func:`get_beta_and_NF`, but returns log normalizing factors. + + The product of child messages is taken in log space, so nodes with thousands of children + (the large polytomies of reconstructed phylogenies) neither overflow nor underflow. + """ # MSD of the respective lineage MSD_array = np.maximum(MSD, np.finfo(MSD.dtype).eps) ELMSD = EL * MSD ### NF leaf calculation - NF = np.zeros(MSD.shape[0], dtype=float) # instantiating N by 1 array - NF[leaves_idx] = np.sum(ELMSD[leaves_idx, :], axis=1) - assert np.all(np.isfinite(NF)) + logNF = np.zeros(MSD.shape[0], dtype=float) # instantiating N by 1 array + NF_leaves = np.sum(ELMSD[leaves_idx, :], axis=1) + assert np.all(np.isfinite(NF_leaves)) and np.all(NF_leaves > 0) + logNF[leaves_idx] = np.log(NF_leaves) ### beta calculation beta = np.zeros_like(MSD) - beta[leaves_idx, :] = ELMSD[leaves_idx, :] / NF[leaves_idx, np.newaxis] + beta[leaves_idx, :] = ELMSD[leaves_idx, :] / NF_leaves[:, np.newaxis] # Assert all ~= 1.0 assert np.abs(np.sum(beta[-1]) - 1.0) < 1e-9 @@ -89,13 +120,20 @@ def get_beta_and_NF( for pii in cIDXs: ch_ii = tree.indices[tree.indptr[pii] : tree.indptr[pii + 1]] - ratt = (beta[ch_ii, :] / MSD_array[ch_ii, :]) @ T.T - fac1 = np.prod(ratt, axis=0) * ELMSD[pii, :] + if T.ndim == 2: + ratt = (beta[ch_ii, :] / MSD_array[ch_ii, :]) @ T.T + else: + ratt = np.einsum("ckl,cl->ck", T[ch_ii], beta[ch_ii, :] / MSD_array[ch_ii, :]) + with np.errstate(divide="ignore"): + log_fac = np.sum(np.log(ratt), axis=0) + np.log(ELMSD[pii, :]) + top = np.max(log_fac) + fac1 = np.exp(log_fac - top) + total = np.sum(fac1) - NF[pii] = np.sum(fac1) - beta[pii, :] = fac1 / NF[pii] + logNF[pii] = top + np.log(total) + beta[pii, :] = fac1 / total - return NF, beta + return logNF, beta def get_gamma( @@ -110,7 +148,7 @@ def get_gamma( x_bar is the observations for the whole tree. :param tree: CSR array representing the lineage tree adjacency - :param T: State transitions matrix + :param T: State transitions matrix, shared (K by K) or per edge (N by K by K) :param MSD: The marginal state distribution P(z_n = k) :param beta: beta values. The conditional probability of states, given observations of the sub-tree rooted in cell_n """ @@ -120,14 +158,25 @@ def get_gamma( epss = np.finfo(np.float32).eps coeffs = beta / np.maximum(MSD, epss) coeffs = np.maximum(coeffs, epss) - beta_parents = T @ coeffs.T - - non_leaves = np.nonzero(np.diff(tree.indptr) > 0)[0] - for pidx in non_leaves: - A_base = gamma[pidx, :].T - for ci in tree.indices[tree.indptr[pidx] : tree.indptr[pidx + 1]]: - A = A_base / beta_parents[:, ci] - gamma[ci, :] = coeffs[ci, :] * (A @ T) + + if T.ndim == 2: + beta_parents = T @ coeffs.T + + non_leaves = np.nonzero(np.diff(tree.indptr) > 0)[0] + for pidx in non_leaves: + A_base = gamma[pidx, :].T + for ci in tree.indices[tree.indptr[pidx] : tree.indptr[pidx + 1]]: + A = A_base / beta_parents[:, ci] + gamma[ci, :] = coeffs[ci, :] * (A @ T) + else: + # beta_parents[c] = T_c @ coeffs[c] for the edge into each cell c + beta_parents = np.einsum("ckl,cl->ck", T, coeffs) + + non_leaves = np.nonzero(np.diff(tree.indptr) > 0)[0] + for pidx in non_leaves: + ch = tree.indices[tree.indptr[pidx] : tree.indptr[pidx + 1]] + A = gamma[pidx, :] / beta_parents[ch, :] + gamma[ch, :] = coeffs[ch, :] * np.einsum("ck,ckl->cl", A, T[ch]) assert np.all(np.isfinite(gamma)) return gamma diff --git a/lineage/HMM/M_step.py b/lineage/HMM/M_step.py index a2553d90e..972cfdd5c 100644 --- a/lineage/HMM/M_step.py +++ b/lineage/HMM/M_step.py @@ -53,3 +53,30 @@ def get_all_zetas( js = gammas[parents, :] / TbetaMSD[daughters, :] holder = np.einsum("ik,il->kl", js, betaMSD[daughters, :]) return holder * T + + +def get_edge_posteriors( + tree: csr_array, + beta_array: npt.NDArray[np.float64], + MSD_array: npt.NDArray[np.float64], + gammas: npt.NDArray[np.float64], + T: npt.NDArray[np.float64], +) -> tuple[np.ndarray, np.ndarray, npt.NDArray[np.float64]]: + """ + Edge-wise joint posterior :math:`P(z_p = k, z_c = l | X)` for every parent-child edge. + + :param T: transition matrix, shared (K by K) or per edge (N by K by K, indexed by the child) + :return: parents, children, and an (edges by K by K) array of joint posteriors; each slice sums to 1. + """ + parents = np.repeat(np.arange(tree.shape[0]), np.diff(tree.indptr)) + daughters = tree.indices + K = beta_array.shape[1] + if tree.nnz == 0: + return parents, daughters, np.zeros((0, K, K)) + + Te = T[daughters] if T.ndim == 3 else np.broadcast_to(T, (len(daughters), K, K)) + betaMSD = beta_array[daughters] / np.clip(MSD_array[daughters], np.finfo(float).eps, np.inf) + TbetaMSD = np.clip(np.einsum("ekl,el->ek", Te, betaMSD), np.finfo(float).eps, np.inf) + js = gammas[parents, :] / TbetaMSD + xi = js[:, :, np.newaxis] * Te * betaMSD[:, np.newaxis, :] + return parents, daughters, xi diff --git a/lineage/LineageTree.py b/lineage/LineageTree.py index d0d1f7b97..beef8dc00 100644 --- a/lineage/LineageTree.py +++ b/lineage/LineageTree.py @@ -26,16 +26,25 @@ class LineageTree: obs: np.ndarray tree: csr_array states: np.ndarray - E: Sequence[StA | StB | StC] + branch_lengths: np.ndarray | None + names: np.ndarray | None + E: Sequence def __init__( self, list_of_cells: list | csr_array, - E: Sequence[StA | StB | StC], + E: Sequence, obs: np.ndarray | None = None, states: np.ndarray | None = None, + branch_lengths: np.ndarray | None = None, ): + """ + :param branch_lengths: Optional length of the edge leading into each cell (0 for the root). + Division-resolved lineages leave this as None and use a single transition matrix per edge. + """ self.E = E + self.branch_lengths = branch_lengths + self.names = None # node names, set when loaded from a named phylogeny if isinstance(list_of_cells, list): self._output_lineage = sorted(list_of_cells, key=operator.attrgetter("gen")) self.tree = lineage_to_tree(self._output_lineage) @@ -179,6 +188,43 @@ def __len__(self): return self.tree.shape[0] +def get_log_Emission_Likelihoods(X: list[LineageTree], E: list) -> list[np.ndarray]: + """ + Log emission likelihood matrix, :math:`\\log P(x_n = x | z_n = k)`, for each lineage. + + Unobserved cells (e.g. internal nodes of a reconstructed phylogeny) should return 0 from ``logpdf``. + + :param X: list of lineage trees + :param E: The emission distributions, one per state + :return: One N by K array per lineage + """ + all_cells = np.vstack([lineage.obs for lineage in X]) + logEL = np.column_stack([E[k].logpdf(all_cells) for k in range(len(E))]) + assert not np.any(np.isnan(logEL)) + splits = np.cumsum([len(lineageObj) for lineageObj in X])[:-1] + return np.split(logEL, splits) + + +def get_scaled_Emission_Likelihoods(X: list[LineageTree], E: list) -> tuple[list[np.ndarray], list[np.ndarray]]: + """ + Emission likelihoods rescaled per cell so that the largest entry in each row is 1. + + High-dimensional observations easily have likelihoods far below the smallest double, so the + per-cell maximum log-likelihood is subtracted before exponentiating. Scaling a cell's row by a + constant leaves beta and gamma unchanged and only scales that cell's normalizing factor, so the + log-likelihood is ``sum(log(NF)) + sum(offsets)``. + + :return: The scaled EL arrays and the per-cell log offsets, one of each per lineage. + """ + EL, offsets = [], [] + for logEL in get_log_Emission_Likelihoods(X, E): + off = np.max(logEL, axis=1) + assert np.all(np.isfinite(off)), "Some cells are impossible under every state." + EL.append(np.exp(logEL - off[:, np.newaxis])) + offsets.append(off) + return EL, offsets + + def get_Emission_Likelihoods(X: list[LineageTree], E: list) -> list[np.ndarray]: """ Emission Likelihood (EL) matrix. @@ -188,25 +234,15 @@ def get_Emission_Likelihoods(X: list[LineageTree], E: list) -> list[np.ndarray]: :math:`P(x_n = x | z_n = k)`, for all :math:`x_n` and :math:`z_n` in our observed and hidden state tree - and for all possible discrete states k. + and for all possible discrete states k. This underflows for high-dimensional + observations; use :func:`get_scaled_Emission_Likelihoods` there. + :param X: list of lineage trees :param E: The emissions likelihood :return: The marginal state distribution """ - all_cells = np.vstack([lineage.obs for lineage in X]) - ELstack = np.zeros((len(all_cells), len(E))) - - for k in range(len(E)): # for each state - ELstack[:, k] = np.exp(E[k].logpdf(all_cells)) - assert np.all(np.isfinite(ELstack[:, k])) - EL = [] - ii = 0 - for lineageObj in X: # for each lineage in our Population - nl = len(lineageObj) # getting the lineage length - EL.append(ELstack[ii : (ii + nl), :]) # append the EL_array for each lineage - - ii += nl - + EL = [np.exp(logEL) for logEL in get_log_Emission_Likelihoods(X, E)] + assert all(np.all(np.isfinite(el)) for el in EL) return EL diff --git a/lineage/ctmc.py b/lineage/ctmc.py new file mode 100644 index 000000000..47f1c75e9 --- /dev/null +++ b/lineage/ctmc.py @@ -0,0 +1,129 @@ +r"""Continuous-time Markov chain transitions on branch-length trees. + +Reconstructed phylogenies (e.g. Cassiopeia trees from CRISPR recorders) have edges that span an +unknown number of divisions, so each edge e gets its own transition matrix +:math:`T(t_e) = \exp(Q t_e)` from a shared rate matrix :math:`Q`. + +The M-step for :math:`Q` uses the endpoint-conditioned expected sufficient statistics of the +chain, following Hobolth & Jensen (2011, "Summary statistics for endpoint-conditioned +continuous-time Markov chains"). With the joint posterior :math:`\xi_e(a, b)` of the states at +both ends of each edge, the expected time spent in state i, :math:`R_i`, and the expected number +of jumps from i to j, :math:`N_{ij}`, are + +.. math:: + + E[R_i] = \sum_e \sum_{ab} \frac{\xi_e(a, b)}{P_{ab}(t_e)} I^{ii}_{ab}(t_e), \qquad + E[N_{ij}] = q_{ij} \sum_e \sum_{ab} \frac{\xi_e(a, b)}{P_{ab}(t_e)} I^{ij}_{ab}(t_e), + +with :math:`I^{ij}(t) = \int_0^t e^{Qs} e_i e_j^\top e^{Q(t-s)} ds`, and the update is +:math:`q_{ij} = E[N_{ij}] / E[R_i]`. With the eigendecomposition :math:`Q = U \Lambda V`, +:math:`V = U^{-1}`, both sums collapse to :math:`(V^\top A U^\top)_{ij}` for +:math:`A = \sum_e (U^\top W_e V^\top) \circ J(t_e)`, :math:`W_e = \xi_e / P(t_e)` and +:math:`J_{cd}(t) = \int_0^t e^{\lambda_c s} e^{\lambda_d (t - s)} ds`. This costs one +eigendecomposition and O(K^2) work per edge. +""" + +import numpy as np + + +def rates_to_Q(rates: np.ndarray) -> np.ndarray: + """Build a rate matrix from its off-diagonal rates (the diagonal of ``rates`` is ignored).""" + Q = np.array(rates, dtype=float) + np.fill_diagonal(Q, 0.0) + np.fill_diagonal(Q, -Q.sum(axis=1)) + return Q + + +def random_Q(K: int, total_rate: float = 1.0, rng=None) -> np.ndarray: + """Random rate matrix whose rows each leave their state at ``total_rate``.""" + rng = np.random.default_rng(rng) + R = rng.dirichlet(np.ones(K - 1), size=K) * total_rate + rates = np.zeros((K, K)) + for i in range(K): + rates[i, np.arange(K) != i] = R[i] + return rates_to_Q(rates) + + +def _eig(Q: np.ndarray) -> tuple[np.ndarray, np.ndarray, np.ndarray]: + """Eigendecomposition Q = U diag(lam) V, with a tiny perturbation if Q is (nearly) defective.""" + lam, U = np.linalg.eig(Q) + if np.linalg.cond(U) > 1e8: + # A defective or nearly-defective Q (e.g. repeated eigenvalues with a shared eigenvector) + # has no stable eigenbasis. Splitting the eigenvalues slightly perturbs expm(Qt) by O(1e-9). + jitter = np.diag(np.linspace(-1.0, 1.0, Q.shape[0])) * 1e-9 * max(1.0, np.abs(Q).max()) + lam, U = np.linalg.eig(Q + jitter) + return lam, U, np.linalg.inv(U) + + +def transition_matrices(Q: np.ndarray, t: np.ndarray) -> np.ndarray: + """:math:`\\exp(Q t)` for every entry of ``t``, shape (len(t), K, K). Rows are clipped and renormalized.""" + t = np.asarray(t, dtype=float) + lam, U, V = _eig(Q) + expo = np.exp(np.outer(t, lam)) # (E, K) + P = np.real(np.einsum("ac,ec,cb->eab", U, expo, V)) + P = np.clip(P, 0.0, None) + P /= P.sum(axis=2, keepdims=True) + return P + + +def _J(lam: np.ndarray, t: np.ndarray) -> np.ndarray: + """:math:`J_{cd}(t) = \\int_0^t e^{\\lambda_c s} e^{\\lambda_d (t-s)} ds` for each t, shape (E, K, K).""" + lc = lam[np.newaxis, :, np.newaxis] + ld = lam[np.newaxis, np.newaxis, :] + tt = t[:, np.newaxis, np.newaxis] + diff = lc - ld + close = np.abs(diff * tt) < 1e-8 + safe = np.where(close, 1.0, diff) + # e^{ld t} (e^{(lc - ld) t} - 1) / (lc - ld), falling back to t e^{ld t} when lc ~ ld + general = np.exp(ld * tt) * np.expm1(diff * tt) / safe + limit = tt * np.exp(0.5 * (lc + ld) * tt) + return np.where(close, limit, general) + + +def expected_statistics(Q: np.ndarray, t: np.ndarray, xi: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + """ + Expected dwell times and jump counts given edge-endpoint posteriors. + + :param Q: current rate matrix (K by K) + :param t: edge lengths (E) + :param xi: joint posteriors of the parent and child states for each edge (E by K by K) + :return: expected time in each state (K) and expected number of i -> j jumps (K by K, zero diagonal) + """ + t = np.asarray(t, dtype=float) + keep = t > 0 # zero-length edges carry no time and no jumps + t, xi = t[keep], xi[keep] + K = Q.shape[0] + if t.size == 0: + return np.zeros(K), np.zeros((K, K)) + + lam, U, V = _eig(Q) + P = transition_matrices(Q, t) + W = xi / np.maximum(P, 1e-300) + A = np.einsum("ac,eab,db->ecd", U, W, V) * _J(lam, t) + S = np.real(V.T @ A.sum(axis=0) @ U.T) + + dwell = np.clip(np.diag(S).copy(), 0.0, None) + jumps = np.clip(Q * S, 0.0, None) + np.fill_diagonal(jumps, 0.0) + return dwell, jumps + + +def M_step_Q( + Q: np.ndarray, t: np.ndarray, xi: np.ndarray, pseudo_jumps: float = 1e-2, pseudo_time: float = 1e-2 +) -> np.ndarray: + """One EM update of the rate matrix. Small pseudocounts keep every rate strictly positive.""" + dwell, jumps = expected_statistics(Q, t, xi) + K = Q.shape[0] + rates = (jumps + pseudo_jumps / (K - 1)) / (dwell + pseudo_time)[:, np.newaxis] + return rates_to_Q(rates) + + +def stationary(Q: np.ndarray) -> np.ndarray: + """Stationary distribution of the chain, solving pi Q = 0.""" + K = Q.shape[0] + A = np.vstack([Q.T, np.ones(K)]) + b = np.zeros(K + 1) + b[-1] = 1.0 + pi = np.linalg.lstsq(A, b, rcond=None)[0] + pi = np.clip(pi, 0.0, None) + return pi / pi.sum() diff --git a/lineage/phyloHMM.py b/lineage/phyloHMM.py new file mode 100644 index 000000000..7d6101949 --- /dev/null +++ b/lineage/phyloHMM.py @@ -0,0 +1,269 @@ +r"""Latent-state tHMM on reconstructed phylogenies with branch-length-dependent transitions. + +Each edge e into node c has transition matrix :math:`T_e = \exp(Q t_e)` (``mode="ctmc"``), or a single +shared matrix T for division-resolved trees (``mode="discrete"``). Emissions are only observed at the +leaves; internal nodes carry NaN observations and emission likelihood 1. The upward-downward passes +reuse :mod:`lineage.HMM.E_step` with per-edge matrices. +""" + +from copy import deepcopy +from dataclasses import dataclass + +import numpy as np +from scipy.special import logsumexp +from sklearn.cluster import KMeans + +from . import ctmc +from .HMM.E_step import get_beta_and_logNF, get_gamma, get_MSD +from .HMM.M_step import get_edge_posteriors +from .LineageTree import LineageTree, get_scaled_Emission_Likelihoods + + +@dataclass +class TreePosterior: + """E-step output for one tree.""" + + gamma: np.ndarray # (N, K) marginal posteriors P(z_n = k | Y) + parents: np.ndarray # (E,) + children: np.ndarray # (E,) + xi: np.ndarray # (E, K, K) joint posteriors P(z_p = k, z_c = l | Y) + LL: float + + +def _lengths(lin: LineageTree) -> np.ndarray: + assert lin.branch_lengths is not None, "ctmc mode needs branch lengths." + return lin.branch_lengths + + +class PhyloHMM: + """Hidden Markov model on a collection of rooted trees with leaf-only observations.""" + + def __init__( + self, + X: list[LineageTree], + num_states: int, + mode: str = "ctmc", + E=None, + rng=None, + fix_Q: np.ndarray | None = None, + fix_E=None, + pi_mode: str = "roots", + ): + """ + :param X: Trees. In ``ctmc`` mode each must have ``branch_lengths``. + :param num_states: Number of hidden states K. + :param mode: ``"ctmc"`` for :math:`\\exp(Q t_e)` per edge, or ``"discrete"`` for one T per edge. + :param E: Template emission object copied for each state (ignored if ``fix_E`` is given). + :param pi_mode: ``"roots"`` estimates the root distribution from the root posteriors; + ``"stationary"`` ties it to the stationary distribution of Q or T. + """ + assert mode in ("ctmc", "discrete") + assert pi_mode in ("roots", "stationary") + self.X = X + self.K = num_states + self.mode = mode + self.pi_mode = pi_mode + self.rng = np.random.default_rng(rng) + if mode == "ctmc": + assert all(x.branch_lengths is not None for x in X), "ctmc mode needs branch lengths." + self.fix_Q = fix_Q + self.fix_E = fix_E + template = E if E is not None else X[0].E[0] + self.E = deepcopy(fix_E) if fix_E is not None else [deepcopy(template) for _ in range(num_states)] + self.pi = np.ones(num_states) / num_states + scale = self._typical_length() + self.Q = fix_Q.copy() if fix_Q is not None else ctmc.random_Q(num_states, 0.5 / scale, self.rng) + self.T = self.rng.dirichlet(np.ones(num_states) * 5, size=num_states) * 0.5 + 0.5 * np.eye(num_states) + self.LL_trace: list[float] = [] + + # ---------------------------------------------------------------- helpers + def _typical_length(self) -> float: + if self.mode != "ctmc": + return 1.0 + bl = np.concatenate([_lengths(x)[1:] for x in self.X]) + bl = bl[bl > 0] + return float(np.median(bl)) if bl.size else 1.0 + + def edge_matrices(self, lin: LineageTree) -> np.ndarray: + """Per-node transition matrices (N, K, K); entry 0 (the root) is unused.""" + if self.mode == "discrete": + return np.broadcast_to(self.T, (len(lin), self.K, self.K)) + return ctmc.transition_matrices(self.Q, _lengths(lin)) + + def observed_leaf_mask(self, lin: LineageTree) -> np.ndarray: + return np.any(np.isfinite(lin.obs), axis=1) + + # ---------------------------------------------------------------- E-step + def e_step(self, X: list[LineageTree] | None = None) -> list[TreePosterior]: + X = self.X if X is None else X + EL, offsets = get_scaled_Emission_Likelihoods(X, self.E) + out = [] + for lin, el, off in zip(X, EL, offsets, strict=True): + T = self.edge_matrices(lin) + MSD = get_MSD(lin.tree, self.pi, T) + logNF, beta = get_beta_and_logNF(lin.leaves_idx, lin.tree, T, MSD, el) + gamma = get_gamma(lin.tree, T, MSD, beta) + parents, children, xi = get_edge_posteriors(lin.tree, beta, MSD, gamma, T) + LL = float(np.sum(logNF) + np.sum(off)) + out.append(TreePosterior(gamma, parents, children, xi, LL)) + return out + + # ---------------------------------------------------------------- M-step + def m_step(self, post: list[TreePosterior]): + K = self.K + if self.fix_E is None: + obs = np.vstack([x.obs for x in self.X]) + gam = np.vstack([p.gamma for p in post]) + for k in range(K): + self.E[k].estimator(obs, gam[:, k]) + + if self.mode == "ctmc": + if self.fix_Q is None: + t = np.concatenate([_lengths(x)[p.children] for x, p in zip(self.X, post, strict=True)]) + xi = np.concatenate([p.xi for p in post]) + self.Q = ctmc.M_step_Q(self.Q, t, xi) + else: + numer = np.full((K, K), 0.1 / K) + sum(p.xi.sum(axis=0) for p in post) + self.T = numer / numer.sum(axis=1, keepdims=True) + + if self.pi_mode == "roots": + roots = np.sum([p.gamma[0] for p in post], axis=0) + 1.0 / K + self.pi = roots / roots.sum() + elif self.mode == "ctmc": + self.pi = ctmc.stationary(self.Q) + else: + w, v = np.linalg.eig(self.T.T) + s = np.real(v[:, np.argmin(np.abs(w - 1))]) + self.pi = s / s.sum() + + # ---------------------------------------------------------------- fitting + def init_emissions(self): + """k-means on the observed leaves, then per-cluster mean and variance.""" + if self.fix_E is not None: + return + obs = np.vstack([x.obs for x in self.X]) + rows = np.all(np.isfinite(obs), axis=1) + km = KMeans(self.K, n_init=1, random_state=int(self.rng.integers(2**31))).fit(obs[rows]) + for k in range(self.K): + w = (km.labels_ == k).astype(float) + self.E[k].estimator(obs[rows], w + 1e-6) + + def fit(self, tol: float = 1e-4, max_iter: int = 300, init: bool = True, verbose: bool = False): + """EM until the per-observed-leaf log-likelihood improves by less than ``tol``.""" + if init: + self.init_emissions() + n_obs = max(1, sum(int(self.observed_leaf_mask(x).sum()) for x in self.X)) + post = self.e_step() + old = sum(p.LL for p in post) + self.LL_trace = [old] + for it in range(max_iter): + self.m_step(post) + post = self.e_step() + new = sum(p.LL for p in post) + self.LL_trace.append(new) + if verbose: + print(it, new) + if (new - old) / n_obs < tol: + break + old = new + self.posteriors = post + self.LL = self.LL_trace[-1] + return self + + # ---------------------------------------------------------------- summaries + def num_parameters(self) -> int: + dof = 0 if self.pi_mode == "stationary" else self.K - 1 + if self.fix_Q is None: + dof += self.K * (self.K - 1) + if self.fix_E is None: + dof += sum(e.dof() for e in self.E) + return dof + + def BIC(self) -> float: + n_obs = sum(int(self.observed_leaf_mask(x).sum()) for x in self.X) + return -2 * self.LL + np.log(n_obs) * self.num_parameters() + + def switch_summary(self, post: list[TreePosterior] | None = None) -> list[dict]: + """Per-tree expected state changes from the edge-wise joint posterior. + + ``edge_changes[k, l]`` is the expected number of edges whose parent is in k and child in l (k != l); + this is the posterior analog of a parsimony count. In ctmc mode ``jumps[k, l]`` additionally counts + every k -> l jump along the branches, including ones that revert within an edge. + """ + post = self.posteriors if post is None else post + out = [] + for lin, p in zip(self.X, post, strict=True): + joint = p.xi.sum(axis=0) + ec = joint.copy() + stay = np.trace(ec) + np.fill_diagonal(ec, 0.0) + per_edge = 1.0 - np.einsum("ekk->e", p.xi) # P(parent state != child state) for each edge + d = { + "n_edges": len(p.children), + "edge_joint": joint, + "edge_changes": ec, + "expected_changes": float(ec.sum()), + "plasticity": float(ec.sum() / max(1, len(p.children))), + "heritability": float(stay / max(1, len(p.children))), + "per_edge_change": per_edge, + } + if self.mode == "ctmc": + t = _lengths(lin)[p.children] + dwell, jumps = ctmc.expected_statistics(self.Q, t, p.xi) + d["jumps"] = jumps + d["dwell"] = dwell + out.append(d) + return out + + def heldout_loglik(self, X_masked: list[LineageTree], X_full: list[LineageTree], lineage: bool = True) -> float: + """Predictive log-density of leaves observed in ``X_full`` but NaN in ``X_masked``, under this fit. + + The posterior at a masked leaf given all other data is exactly its predictive state distribution, + so the score is :math:`\\sum_j \\log \\sum_k P(z_j = k | Y_{-j}) p(y_j | z_j = k)`. + With ``lineage=False`` the tree is ignored at prediction time (population state frequencies + replace the posterior), so the difference between the two scores is the information the + lineage carries about a cell's state. + """ + post = self.e_step(X_masked) + if not lineage: + # Same emissions, but every masked cell gets the population-wide state frequencies + # of the observed leaves: what the embedding alone predicts without the tree. + obs_gam = np.vstack([p.gamma[self.observed_leaf_mask(x)] for x, p in zip(X_masked, post, strict=True)]) + freq = obs_gam.mean(axis=0) + total = 0.0 + for lin_m, lin_f, p in zip(X_masked, X_full, post, strict=True): + hidden = self.observed_leaf_mask(lin_f) & ~self.observed_leaf_mask(lin_m) + if not np.any(hidden): + continue + logEL = np.column_stack([e.logpdf(lin_f.obs[hidden]) for e in self.E]) + w = p.gamma[hidden] if lineage else np.broadcast_to(freq, logEL.shape) + total += float(np.sum(logsumexp(logEL + np.log(np.maximum(w, 1e-300)), axis=1))) + return total + + +def fit_best(X, num_states, n_init=5, rng=None, **kwargs) -> PhyloHMM: + """Fit from several random starts and keep the highest log-likelihood.""" + rng = np.random.default_rng(rng) + fit_kwargs = {k: kwargs.pop(k) for k in ("tol", "max_iter") if k in kwargs} + best = None + for _ in range(n_init): + m = PhyloHMM(X, num_states, rng=rng, **kwargs).fit(**fit_kwargs) + if best is None or m.LL > best.LL: + best = m + assert best is not None + return best + + +def mask_leaves(X: list[LineageTree], frac: float, rng=None) -> list[LineageTree]: + """Copy the trees with a random fraction of the observed leaves set to NaN.""" + rng = np.random.default_rng(rng) + out = [] + for lin in X: + obs = lin.obs.copy() + observed = np.nonzero(np.any(np.isfinite(obs), axis=1))[0] + hide = observed[rng.random(observed.size) < frac] + obs[hide] = np.nan + new = LineageTree(lin.tree, lin.E, obs=obs, branch_lengths=lin.branch_lengths) + new.names = lin.names + out.append(new) + return out diff --git a/lineage/phylo_sim.py b/lineage/phylo_sim.py new file mode 100644 index 000000000..aac54b8d5 --- /dev/null +++ b/lineage/phylo_sim.py @@ -0,0 +1,96 @@ +"""Simulate branch-length trees with CTMC state changes and Gaussian leaf emissions.""" + +import networkx as nx +import numpy as np +from scipy.linalg import expm + +from .tree_io import PhyloTree, graph_to_phylotree, prune_graph + + +def yule_tree(n_leaves: int, birth_rate: float = 1.0, rng=None) -> nx.DiGraph: + """Pure-birth tree grown until ``n_leaves`` lineages exist; branch lengths are in time units. + + Every leaf is sampled at the same final time (the moment the last split happens plus an + exponential waiting time), as in a tumor harvested at one time point. + """ + rng = np.random.default_rng(rng) + g = nx.DiGraph() + g.add_node("n0", time=0.0) + active = ["n0"] + t = 0.0 + counter = 1 + while len(active) < n_leaves: + t += rng.exponential(1.0 / (birth_rate * len(active))) + parent = active.pop(int(rng.integers(len(active)))) + g.nodes[parent]["time_end"] = t + for _ in range(2): + child = f"n{counter}" + counter += 1 + g.add_node(child, time=t) + active.append(child) + g.add_edge(parent, child) + t_end = t + rng.exponential(1.0 / (birth_rate * len(active))) + for n in active: + g.nodes[n]["time_end"] = t_end + # A node's edge length is the time from its parent's split to its own split (or sampling) + for u, v in g.edges: + g[u][v]["length"] = g.nodes[v]["time_end"] - g.nodes[v]["time"] + # The root lineage lives from 0 until its split; fold that stem into nothing (root state is at its split) + return g + + +def simulate_states(pt: PhyloTree, Q: np.ndarray, pi: np.ndarray, rng=None) -> np.ndarray: + """Sample node states from the root distribution and exp(Q t) along each edge.""" + rng = np.random.default_rng(rng) + K = Q.shape[0] + states = np.zeros(len(pt.names), dtype=int) + states[0] = rng.choice(K, p=pi) + par = pt.parents + cache: dict[float, np.ndarray] = {} + for i in range(1, len(states)): + t = float(pt.branch_lengths[i]) + if t not in cache: + P = np.clip(expm(Q * t), 0, None) + cache[t] = P / P.sum(axis=1, keepdims=True) + states[i] = rng.choice(K, p=cache[t][states[par[i]]]) + return states + + +def simulate_dataset( + n_trees: int, + n_leaves: int, + Q: np.ndarray, + pi: np.ndarray, + means: np.ndarray, + sd: float | np.ndarray, + sample_frac: float = 1.0, + rng=None, +) -> list[dict]: + """Simulate trees, states and leaf observations, then subsample leaves. + + Subsampled trees are pruned to the retained leaves with unifurcations collapsed (branch lengths + summed), as for a real recorder experiment that sequences a fraction of the tumor. + + :return: one dict per tree with the full and sampled PhyloTrees, the true states of the sampled + tree's nodes (internal nodes of the pruned tree are real ancestors), and leaf observations. + """ + rng = np.random.default_rng(rng) + K, D = means.shape + sd = np.broadcast_to(np.asarray(sd, dtype=float), (K, D)) + out = [] + for _ in range(n_trees): + g = yule_tree(n_leaves, rng=rng) + full = graph_to_phylotree(g) + states = simulate_states(full, Q, pi, rng) + state_of = dict(zip(full.names, states, strict=True)) + leaves = full.leaf_names + keep = leaves[rng.random(leaves.size) < sample_frac] if sample_frac < 1 else leaves + if keep.size < 3: + keep = rng.choice(leaves, size=3, replace=False) + sampled = graph_to_phylotree(prune_graph(g, keep, collapse_unifurcations=True)) + s_states = np.array([state_of[n] for n in sampled.names]) + obs = np.full((len(sampled.names), D), np.nan) + leaf = sampled.is_leaf + obs[leaf] = rng.normal(means[s_states[leaf]], sd[s_states[leaf]]) + out.append({"full": full, "tree": sampled, "states": s_states, "obs": obs}) + return out diff --git a/lineage/phylo_stats.py b/lineage/phylo_stats.py new file mode 100644 index 000000000..0fe740b53 --- /dev/null +++ b/lineage/phylo_stats.py @@ -0,0 +1,90 @@ +"""Baselines that label the leaves first and then measure how the labels relate to the tree. + +* :func:`small_parsimony` is unit-cost Sankoff parsimony, which equals Fitch-Hartigan on + multifurcating trees and is what Cassiopeia's FitchCount builds on. +* :func:`phylogenetic_correlation` is the Moran's-I phylogenetic auto/cross-correlation used by + PATH (Schiffman et al. 2024), with inverse node-distance weights. +""" + +import numpy as np +from scipy.sparse.csgraph import shortest_path + +from .tree_io import PhyloTree + + +def small_parsimony(pt: PhyloTree, leaf_labels: np.ndarray, K: int) -> tuple[int, np.ndarray, np.ndarray]: + """Minimum number of label changes, one optimal labeling of every node, and its transition counts. + + :param leaf_labels: integer label per node (only leaves are read); -1 marks an unlabeled leaf. + :return: parsimony score, per-node labels, and a K by K matrix counting parent -> child changes. + """ + N = len(pt.names) + tree = pt.tree + cost = np.zeros((N, K)) + leaf = pt.is_leaf + for i in np.nonzero(leaf)[0]: + if leaf_labels[i] >= 0: + cost[i] = np.inf + cost[i, leaf_labels[i]] = 0.0 + change = 1.0 - np.eye(K) + for p in np.nonzero(~leaf)[0][::-1]: + ch = tree.indices[tree.indptr[p] : tree.indptr[p + 1]] + # min over the child's state of (child cost + change cost), for each parent state + cost[p] = np.sum(np.min(cost[ch][:, np.newaxis, :] + change[np.newaxis], axis=2), axis=0) + + labels = np.full(N, -1) + labels[0] = int(np.argmin(cost[0])) + counts = np.zeros((K, K), dtype=int) + for p in np.nonzero(~leaf)[0]: + for c in tree.indices[tree.indptr[p] : tree.indptr[p + 1]]: + opt = cost[c] + change[labels[p]] + # Prefer keeping the parent's label among ties, as Fitch does + best = labels[p] if opt[labels[p]] <= opt.min() else int(np.argmin(opt)) + labels[c] = best + counts[labels[p], best] += 1 + np.fill_diagonal(counts, 0) + return int(counts.sum()), labels, counts + + +def leaf_node_distance(pt: PhyloTree, leaves: np.ndarray | None = None, weighted: bool = False) -> np.ndarray: + """Pairwise path lengths between leaves, in edges (or branch lengths if ``weighted``).""" + leaves = np.nonzero(pt.is_leaf)[0] if leaves is None else leaves + A = pt.tree.astype(float) + if weighted: + A = A.multiply(pt.branch_lengths[np.newaxis, :]).tocsr() + return shortest_path(A, directed=False, indices=leaves)[:, leaves] + + +def _inverse_distance_weights(dist: np.ndarray) -> np.ndarray: + W = np.zeros_like(dist, dtype=float) + off = dist > 0 + W[off] = 1.0 / dist[off] + return W / W.sum() + + +def _moran(W: np.ndarray, labels: np.ndarray, K: int) -> np.ndarray: + onehot = np.eye(K)[labels] + Z = onehot - onehot.mean(axis=0) + norm = np.sqrt(np.sum(Z**2, axis=0)) + norm[norm == 0] = np.inf + return labels.size * (Z.T @ W @ Z) / np.outer(norm, norm) + + +def phylogenetic_correlation(labels: np.ndarray, dist: np.ndarray, K: int) -> np.ndarray: + """PATH-style Moran's I between one-hot state indicators, with weights :math:`w_{ij} = 1/d_{ij}`. + + Diagonal entries are the phylogenetic auto-correlations (heritability of each state); off-diagonal + entries are cross-correlations (positive means the two states sit in nearby clades). + """ + return _moran(_inverse_distance_weights(dist), labels, K) + + +def phylogenetic_correlation_z(labels: np.ndarray, dist: np.ndarray, K: int, n_perm: int = 100, rng=None): + """Moran's I (as :func:`phylogenetic_correlation`) and its z-score against leaf-label permutations.""" + rng = np.random.default_rng(rng) + W = _inverse_distance_weights(dist) + obs = _moran(W, labels, K) + perms = np.array([_moran(W, rng.permutation(labels), K) for _ in range(n_perm)]) + sd = perms.std(axis=0) + sd[sd == 0] = np.inf + return obs, (obs - perms.mean(axis=0)) / sd diff --git a/lineage/states/StateDistributionGaussian.py b/lineage/states/StateDistributionGaussian.py new file mode 100644 index 000000000..355c972bb --- /dev/null +++ b/lineage/states/StateDistributionGaussian.py @@ -0,0 +1,75 @@ +"""Multivariate Gaussian emissions for continuous per-cell embeddings (e.g. PCA or scVI latents).""" + +import numpy as np + + +class StateDistribution: + r""" + Gaussian emission with diagonal-plus-ridge covariance, :math:`\Sigma_k = \mathrm{diag}(\sigma_k^2) + \epsilon I`. + + Rows of ``x`` that are entirely NaN are treated as unobserved (e.g. the internal nodes of a + reconstructed phylogeny, or held-out leaves) and have log-likelihood 0, so their emission + likelihood is 1. Individually missing dimensions are marginalized out. + """ + + def __init__(self, mean: np.ndarray | None = None, var: np.ndarray | None = None, ridge: float = 1e-3, dim=None): + if mean is None: + assert dim is not None, "Either a mean or the dimension must be given." + mean = np.zeros(dim) + mean = np.asarray(mean, dtype=float) + var = np.ones_like(mean) if var is None else np.asarray(var, dtype=float) + assert mean.shape == var.shape + self.ridge = ridge + self.params = np.concatenate([mean, var]) + + @property + def dim(self) -> int: + return self.params.size // 2 + + @property + def mean(self) -> np.ndarray: + return self.params[: self.dim] + + @property + def var(self) -> np.ndarray: + """The ridge is carried inside the stored variance, so this is the full diagonal of :math:`\\Sigma_k`.""" + return self.params[self.dim :] + + def rvs(self, size: int, rng=None) -> tuple[np.ndarray, ...]: + """Draw ``size`` observations, returned one array per dimension like the other emission classes.""" + rng = np.random.default_rng(rng) + draws = rng.normal(self.mean, np.sqrt(self.var), size=(size, self.dim)) + return tuple(draws.T) + + def dist(self, other) -> float: + """Euclidean distance between the state means.""" + assert isinstance(self, type(other)) + return float(np.linalg.norm(self.mean - other.mean)) + + def dof(self) -> int: + """A mean and a variance per dimension.""" + return 2 * self.dim + + def logpdf(self, x: np.ndarray) -> np.ndarray: + """Log-density of each row, marginalizing out NaN entries. All-NaN rows return 0.""" + x = np.atleast_2d(np.asarray(x, dtype=float)) + observed = np.isfinite(x) + resid = np.where(observed, x - self.mean, 0.0) + per_dim = -0.5 * (np.log(2 * np.pi * self.var) + resid**2 / self.var) + return np.sum(np.where(observed, per_dim, 0.0), axis=1) + + def estimator(self, x: np.ndarray, gammas: np.ndarray): + """Weighted maximum-likelihood mean and variance. Unobserved entries carry no weight.""" + x = np.atleast_2d(np.asarray(x, dtype=float)) + observed = np.isfinite(x) + w = gammas[:, np.newaxis] * observed + wsum = np.sum(w, axis=0) + if np.any(wsum <= 1e-12): + # A state with (almost) no responsibility keeps its old parameters on those dimensions. + wsum = np.where(wsum <= 1e-12, np.nan, wsum) + xz = np.where(observed, x, 0.0) + mean = np.sum(w * xz, axis=0) / wsum + var = np.sum(w * (xz - np.nan_to_num(mean)) ** 2, axis=0) / wsum + self.ridge + mean = np.where(np.isfinite(mean), mean, self.mean) + var = np.where(np.isfinite(var), var, self.var) + self.params = np.concatenate([mean, var]) diff --git a/lineage/tests/test_phyloHMM.py b/lineage/tests/test_phyloHMM.py new file mode 100644 index 000000000..bdd130d73 --- /dev/null +++ b/lineage/tests/test_phyloHMM.py @@ -0,0 +1,318 @@ +"""Tests for the branch-length tHMM on reconstructed phylogenies.""" + +import itertools + +import numpy as np +import pytest +import scipy.stats as sp +from scipy.integrate import quad_vec +from scipy.linalg import expm + +from .. import ctmc +from ..HMM.E_step import get_beta_and_NF, get_gamma, get_MSD +from ..HMM.M_step import get_all_zetas, get_edge_posteriors +from ..LineageTree import LineageTree, get_scaled_Emission_Likelihoods +from ..phylo_sim import simulate_dataset +from ..phyloHMM import PhyloHMM, mask_leaves +from ..states.StateDistributionGaussian import StateDistribution as Gaussian +from ..tree_io import ( + align_observations, + graph_to_phylotree, + lca_character_states, + load_tree, + mutation_branch_lengths, + newick_to_graph, + to_lineage, +) + +NEWICK = "((a:1,b:2)x:0.5,(c:1,(d:1)u:2,e:0.5):1)r;" + + +def small_lineage(D=2, rng=0): + """Root with a binary and a ternary child subtree (one unifurcation collapsed); leaves observed.""" + rng = np.random.default_rng(rng) + pt = load_tree(NEWICK) + obs = rng.normal(size=(5, D)) + return pt, to_lineage(pt, ["a", "b", "c", "d", "e"], obs, [Gaussian(dim=D)]) + + +def brute_force(model: PhyloHMM, lin: LineageTree): + """Log-likelihood, node marginals and edge joints by enumerating every state assignment.""" + K, N = model.K, len(lin) + T = model.edge_matrices(lin) + logEL = np.column_stack([e.logpdf(lin.obs) for e in model.E]) + parents, children = lin.edges + logw, marg, joint = [], np.zeros((N, K)), np.zeros((len(children), K, K)) + for z in itertools.product(range(K), repeat=N): + z = np.array(z) + lw = np.log(model.pi[z[0]]) + np.sum(np.log(T[children, z[parents], z[children]])) + lw += np.sum(logEL[np.arange(N), z]) + logw.append((lw, z)) + m = max(lw for lw, _ in logw) + total = sum(np.exp(lw - m) for lw, _ in logw) + for lw, z in logw: + p = np.exp(lw - m) / total + marg[np.arange(N), z] += p + joint[np.arange(len(children)), z[parents], z[children]] += p + return m + np.log(total), marg, joint + + +# ---------------------------------------------------------------------------- emissions + + +def test_gaussian_logpdf_and_missing(): + g = Gaussian(mean=np.array([0.5, -1.0]), var=np.array([2.0, 0.5])) + x = np.array([[0.1, 0.2], [np.nan, 0.3], [np.nan, np.nan]]) + ll = g.logpdf(x) + ref = sp.norm.logpdf(0.1, 0.5, np.sqrt(2)) + sp.norm.logpdf(0.2, -1, np.sqrt(0.5)) + assert ll[0] == pytest.approx(ref) + assert ll[1] == pytest.approx(sp.norm.logpdf(0.3, -1, np.sqrt(0.5))) + assert ll[2] == 0.0 + + +def test_gaussian_estimator_ignores_unobserved(): + rng = np.random.default_rng(0) + x = rng.normal([3.0, -2.0], [1.0, 0.5], size=(4000, 2)) + x = np.vstack([x, np.full((100, 2), np.nan)]) + g = Gaussian(dim=2, ridge=0.0) + g.estimator(x, np.ones(x.shape[0])) + np.testing.assert_allclose(g.mean, [3.0, -2.0], atol=0.05) + np.testing.assert_allclose(g.var, [1.0, 0.25], rtol=0.1) + + +def test_scaled_emissions_do_not_underflow(): + """With D = 300 the raw likelihoods are exp(-400) or smaller; the scaled ones stay finite.""" + rng = np.random.default_rng(1) + pt, lin = small_lineage(D=300) + lin.obs[pt.is_leaf] = rng.normal(0, 3, size=(5, 300)) + E = [Gaussian(dim=300), Gaussian(mean=np.ones(300))] + EL, off = get_scaled_Emission_Likelihoods([lin], E) + assert np.all(np.isfinite(EL[0])) and np.all(EL[0].max(axis=1) == 1.0) + assert np.all(off[0][~pt.is_leaf] == 0.0) + assert np.min(off[0][pt.is_leaf]) < -700 + + +# ---------------------------------------------------------------------------- E-step + + +def test_per_edge_matches_shared_T(): + """Passing the same T per edge gives identical MSD, beta, gamma and zetas as one shared T.""" + rng = np.random.default_rng(2) + _, lin = small_lineage() + K = 3 + T = rng.dirichlet(np.ones(K), size=K) + pi = rng.dirichlet(np.ones(K)) + EL = rng.random((len(lin), K)) + Tn = np.broadcast_to(T, (len(lin), K, K)).copy() + res = [] + for TT in (T, Tn): + MSD = get_MSD(lin.tree, pi, TT) + NF, beta = get_beta_and_NF(lin.leaves_idx, lin.tree, TT, MSD, EL) + gamma = get_gamma(lin.tree, TT, MSD, beta) + res.append((MSD, NF, beta, gamma)) + for a, b in zip(*res, strict=True): + np.testing.assert_allclose(a, b) + MSD, _, beta, gamma = res[0] + _, _, xi = get_edge_posteriors(lin.tree, beta, MSD, gamma, Tn) + np.testing.assert_allclose(xi.sum(axis=0), get_all_zetas(lin.tree, beta, MSD, gamma, T)) + + +@pytest.mark.parametrize("mode", ["ctmc", "discrete"]) +def test_estep_matches_enumeration(mode): + """Log-likelihood, marginals and edge joints agree with brute-force enumeration on a multifurcating tree.""" + rng = np.random.default_rng(3) + _, lin = small_lineage(D=2) + m = PhyloHMM([lin], 2, mode=mode, E=Gaussian(dim=2), rng=4) + m.E = [Gaussian(mean=rng.normal(size=2)), Gaussian(mean=rng.normal(size=2), var=np.array([0.5, 2.0]))] + m.pi = np.array([0.3, 0.7]) + m.Q = ctmc.rates_to_Q(np.array([[0, 0.4], [0.9, 0]])) + post = m.e_step()[0] + LL, marg, joint = brute_force(m, lin) + assert post.LL == pytest.approx(LL) + np.testing.assert_allclose(post.gamma, marg, atol=1e-10) + np.testing.assert_allclose(post.xi, joint, atol=1e-10) + + +# ---------------------------------------------------------------------------- CTMC + + +def test_ctmc_statistics_match_quadrature(): + rng = np.random.default_rng(5) + K = 3 + Q = ctmc.random_Q(K, 1.2, rng) + t = np.array([0.4, 1.3]) + xi = rng.dirichlet(np.ones(K * K), size=2).reshape(2, K, K) + dwell, jumps = ctmc.expected_statistics(Q, t, xi) + ref_d, ref_n = np.zeros(K), np.zeros((K, K)) + for e in range(2): + P = expm(Q * t[e]) + for i, j in itertools.product(range(K), repeat=2): + integrand = lambda s, i=i, j=j, e=e: np.outer(expm(Q * s)[:, i], expm(Q * (t[e] - s))[j, :]) # noqa: E731 + val = np.sum(xi[e] * quad_vec(integrand, 0, t[e])[0] / P) + if i == j: + ref_d[i] += val + else: + ref_n[i, j] += Q[i, j] * val + np.testing.assert_allclose(dwell, ref_d, atol=1e-10) + np.testing.assert_allclose(jumps, ref_n, atol=1e-10) + assert dwell.sum() == pytest.approx(t.sum()) + np.testing.assert_allclose(ctmc.transition_matrices(Q, t)[1], expm(Q * 1.3), atol=1e-12) + + +def test_ctmc_repeated_eigenvalues(): + Q = ctmc.rates_to_Q(np.ones((3, 3))) + np.testing.assert_allclose(ctmc.transition_matrices(Q, np.array([0.7]))[0], expm(Q * 0.7), atol=1e-8) + dwell, _ = ctmc.expected_statistics(Q, np.array([0.7]), np.full((1, 3, 3), 1 / 9)) + assert dwell.sum() == pytest.approx(0.7) + + +def test_stationary(): + Q = ctmc.random_Q(4, 1.0, 6) + pi = ctmc.stationary(Q) + np.testing.assert_allclose(pi @ Q, 0, atol=1e-12) + + +# ---------------------------------------------------------------------------- tree loading + + +def test_newick_loader_prunes_and_collapses(): + pt = load_tree(NEWICK) + assert pt.names[0] == "r" + # d's unifurcating parent u is merged: d hangs off the ternary node with length 2 + 1 + assert "u" not in set(pt.names) + d = int(np.nonzero(pt.names == "d")[0][0]) + assert pt.branch_lengths[d] == pytest.approx(3.0) + # Parents come before children, and the last node is a leaf + par = pt.parents + assert np.all(par[1:] < np.arange(1, len(par))) + assert pt.is_leaf[-1] + assert sorted(pt.leaf_names) == ["a", "b", "c", "d", "e"] + + pruned = load_tree(NEWICK, keep_leaves=["a", "c", "e"]) + assert sorted(pruned.leaf_names) == ["a", "c", "e"] + a = int(np.nonzero(pruned.names == "a")[0][0]) + assert pruned.branch_lengths[a] == pytest.approx(1.5) # x collapsed into a + + +def test_graph_roundtrip_and_unnamed_nodes(): + g = newick_to_graph("((a:1,b:1):1,c:2);") + assert len(g) == 5 + pt = graph_to_phylotree(g) + g2 = pt.to_graph() + assert set(g2.edges) == set(g.edges) + + +def test_align_observations(): + pt = load_tree(NEWICK) + obs = align_observations(pt, ["e", "a", "zz"], np.array([[5.0], [1.0], [9.0]])) + assert obs[pt.names == "a"][0, 0] == 1.0 + assert obs[pt.names == "e"][0, 0] == 5.0 + assert np.isnan(obs[pt.names == "b"][0, 0]) + assert np.all(np.isnan(obs[~pt.is_leaf])) + + +def test_lca_characters_and_mutation_lengths(): + pt = load_tree("((a,b)x,c)r;") + chars = np.zeros((len(pt.names), 3), dtype=int) + rows = {"a": [1, 2, -1], "b": [1, 3, 4], "c": [0, 2, 4]} + for n, v in rows.items(): + chars[pt.names == n] = v + anc = lca_character_states(pt, chars) + np.testing.assert_array_equal(anc[pt.names == "x"][0], [1, 0, 4]) + np.testing.assert_array_equal(anc[pt.names == "r"][0], [0, 0, 4]) + bl = mutation_branch_lengths(pt, anc) + assert bl[pt.names == "x"][0] == 1 # site 0 cut + assert bl[pt.names == "a"][0] == 1 # site 1 cut; site 2 missing + assert bl[pt.names == "c"][0] == 1 + + +# ---------------------------------------------------------------------------- fitting + + +def simulated(sample_frac=1.0, rng=0): + rng = np.random.default_rng(rng) + Q = ctmc.rates_to_Q(np.array([[0, 0.3, 0.05], [0.1, 0, 0.2], [0.02, 0.05, 0]])) + pi = np.array([0.8, 0.15, 0.05]) + means = rng.normal(0, 1.5, (3, 4)) + data = simulate_dataset(8, 150, Q, pi, means, 1.0, sample_frac=sample_frac, rng=rng) + X = [to_lineage(d["tree"], d["tree"].leaf_names, d["obs"][d["tree"].is_leaf], [Gaussian(dim=4)]) for d in data] + return X, data, Q, means + + +def test_fit_monotone_and_recovers_Q(): + X, _, Q, means = simulated() + m = PhyloHMM(X, 3, rng=1).fit(tol=1e-6) + assert np.all(np.diff(m.LL_trace) > -1e-6), "EM must not decrease the likelihood" + order = [int(np.argmin([np.linalg.norm(e.mean - mu) for e in m.E])) for mu in means] + assert sorted(order) == [0, 1, 2] + Qhat = m.Q[np.ix_(order, order)] + np.testing.assert_allclose(Qhat, Q, atol=0.12) + summ = m.switch_summary() + assert all(0 <= s["plasticity"] <= 1 for s in summ) + assert all(s["jumps"].sum() >= s["expected_changes"] - 1e-8 for s in summ) + + +def test_heldout_loglik(): + X, _, _, _ = simulated(rng=2) + Xm = mask_leaves(X, 0.3, rng=0) + m = PhyloHMM(Xm, 3, rng=0).fit(max_iter=30) + ll = m.heldout_loglik(Xm, X) + assert np.isfinite(ll) and ll < 0 + assert np.isfinite(m.BIC()) + + +def test_discrete_mode_fits(): + X, _, _, _ = simulated(rng=3) + for x in X: + x.branch_lengths = None + m = PhyloHMM(X, 3, mode="discrete", rng=0).fit(max_iter=30) + np.testing.assert_allclose(m.T.sum(axis=1), 1.0) + assert np.all(np.diff(m.LL_trace) > -1e-6) + + +# ---------------------------------------------------------------------------- baselines + + +def test_small_parsimony_multifurcation(): + from ..phylo_stats import small_parsimony + + pt = load_tree("((a,b,c)x,(d,e)y)r;") + lab = np.full(len(pt.names), -1) + for n, v in {"a": 0, "b": 0, "c": 1, "d": 1, "e": 1}.items(): + lab[pt.names == n] = v + score, labels, counts = small_parsimony(pt, lab, 2) + assert score == 2 # x is 0 (one change to c), and r -> y is one change + assert counts.sum() == score + assert labels[pt.names == "x"][0] == 0 and labels[pt.names == "y"][0] == 1 + + +def test_phylogenetic_correlation_signs(): + from ..phylo_stats import leaf_node_distance, phylogenetic_correlation + + pt = load_tree("(((a,b),(c,d)),((e,f),(g,h)));") + dist = leaf_node_distance(pt) + clustered = np.array([0 if n in "abcd" else 1 for n in pt.leaf_names]) + mixed = np.array([0 if n in "aceg" else 1 for n in pt.leaf_names]) + assert phylogenetic_correlation(clustered, dist, 2)[0, 0] > 0 + assert phylogenetic_correlation(mixed, dist, 2)[0, 0] < phylogenetic_correlation(clustered, dist, 2)[0, 0] + + +def test_large_polytomy_is_stable(): + """A root with 5000 leaf children (as in saturated recorder trees) must not overflow the upward pass.""" + n = 5000 + newick = "(" + ",".join(f"c{i}:1" for i in range(n)) + ")r;" + pt = load_tree(newick) + rng = np.random.default_rng(0) + lin = to_lineage(pt, pt.leaf_names, rng.normal(size=(n, 3)), [Gaussian(dim=3)]) + m = PhyloHMM([lin], 3, rng=0) + m.E = [Gaussian(mean=np.full(3, v)) for v in (-1.0, 0.0, 1.0)] + post = m.e_step()[0] + assert np.isfinite(post.LL) + np.testing.assert_allclose(post.gamma.sum(axis=1), 1.0) + # With a single internal node the likelihood factorizes over the root state + logEL = np.column_stack([e.logpdf(lin.obs[1:]) for e in m.E]) + T = ctmc.transition_matrices(m.Q, np.array([1.0]))[0] + from scipy.special import logsumexp + + per_root = np.array([np.sum(logsumexp(np.log(T[k])[None, :] + logEL, axis=1)) for k in range(3)]) + assert post.LL == pytest.approx(logsumexp(per_root + np.log(m.pi))) diff --git a/lineage/tree_io.py b/lineage/tree_io.py new file mode 100644 index 000000000..947834732 --- /dev/null +++ b/lineage/tree_io.py @@ -0,0 +1,239 @@ +"""Load reconstructed phylogenies (Newick, networkx, or Cassiopeia) into the CSR form used by LineageTree.""" + +from collections import deque +from collections.abc import Iterable, Sequence +from dataclasses import dataclass +from io import StringIO +from pathlib import Path + +import networkx as nx +import numpy as np +from Bio import Phylo +from scipy.sparse import csr_array + +from .LineageTree import LineageTree + + +@dataclass +class PhyloTree: + """A rooted tree in breadth-first order: the root is index 0 and parents precede their children.""" + + tree: csr_array + names: np.ndarray + branch_lengths: np.ndarray # length of the edge into each node; 0 for the root + + @property + def is_leaf(self) -> np.ndarray: + return np.diff(self.tree.indptr) == 0 + + @property + def leaf_names(self) -> np.ndarray: + return self.names[self.is_leaf] + + @property + def parents(self) -> np.ndarray: + """Parent index of each node (-1 for the root).""" + par = np.full(len(self.names), -1) + par[self.tree.indices] = np.repeat(np.arange(len(self.names)), np.diff(self.tree.indptr)) + return par + + def depth(self) -> np.ndarray: + """Sum of branch lengths from the root to each node.""" + d = np.zeros(len(self.names)) + par = self.parents + for i in range(1, len(d)): # BFS order, so the parent is already filled in + d[i] = d[par[i]] + self.branch_lengths[i] + return d + + def to_graph(self) -> nx.DiGraph: + g = nx.DiGraph() + g.add_nodes_from(self.names) + par = self.parents + for i in range(1, len(self.names)): + g.add_edge(self.names[par[i]], self.names[i], length=float(self.branch_lengths[i])) + return g + + +def newick_to_graph(newick: str | Path) -> nx.DiGraph: + """Parse a Newick file or string into a DiGraph with a ``length`` attribute on each edge. + + Unnamed internal nodes are named ``node{i}``. Missing branch lengths are recorded as NaN. + """ + text = Path(newick).read_text() if _is_file(newick) else str(newick) + phylo = Phylo.read(StringIO(text.strip()), "newick") + g = nx.DiGraph() + counter = 0 + + def name_of(clade) -> str: + nonlocal counter + if clade.name is None or clade.name == "": + clade.name = f"node{counter}" + counter += 1 + return clade.name + + root_name = name_of(phylo.root) + g.add_node(root_name) + stack = [phylo.root] + while stack: + clade = stack.pop() + for child in clade.clades: + bl = np.nan if child.branch_length is None else float(child.branch_length) + g.add_edge(clade.name, name_of(child), length=bl) + stack.append(child) + assert nx.is_arborescence(g), "The Newick string does not describe a rooted tree with unique node names." + return g + + +def _is_file(x) -> bool: + if isinstance(x, Path): + return True + try: + return Path(str(x)).is_file() + except OSError: # a long Newick string is not a valid path + return False + + +def cassiopeia_to_graph(ctree) -> nx.DiGraph: + """Convert a ``cassiopeia.data.CassiopeiaTree`` without importing Cassiopeia.""" + g = nx.DiGraph() + for u, v in ctree.edges: + g.add_edge(u, v, length=float(ctree.get_branch_length(u, v))) + return g + + +def prune_graph(g: nx.DiGraph, keep_leaves: Iterable[str], collapse_unifurcations: bool = True) -> nx.DiGraph: + """Keep only the given leaves (and their ancestors), then merge single-child nodes into their child. + + Branch lengths are summed through collapsed nodes. The root is kept even if it has one child, unless + that child is the only path to the retained leaves and the root itself would then be a unifurcation. + """ + g = g.copy() + keep = set(keep_leaves) + leaves = [n for n in g if g.out_degree(n) == 0] + g.remove_nodes_from([n for n in leaves if n not in keep]) + # Remove internal nodes left without descendants + while True: + dead = [n for n in g if g.out_degree(n) == 0 and n not in keep] + if not dead: + break + g.remove_nodes_from(dead) + + if collapse_unifurcations: + root = next(n for n in g if g.in_degree(n) == 0) + # Collapse a unifurcating root into its child + while g.out_degree(root) == 1: + (child,) = g.successors(root) + g.remove_node(root) + root = child + for n in list(nx.dfs_postorder_nodes(g, root)): + if n != root and g.out_degree(n) == 1: + (parent,) = g.predecessors(n) + (child,) = g.successors(n) + length = _length(g, parent, n) + _length(g, n, child) + g.remove_node(n) + g.add_edge(parent, child, length=length) + return g + + +def _length(g: nx.DiGraph, u, v) -> float: + return float(g[u][v].get("length", np.nan)) + + +def graph_to_phylotree(g: nx.DiGraph) -> PhyloTree: + """Order nodes breadth-first from the root and build the parent-to-child CSR adjacency.""" + roots = [n for n in g if g.in_degree(n) == 0] + assert len(roots) == 1, "The graph must have exactly one root." + order = [] + q = deque(roots) + while q: + n = q.popleft() + order.append(n) + q.extend(sorted(g.successors(n), key=str)) + idx = {n: i for i, n in enumerate(order)} + N = len(order) + indptr = np.zeros(N + 1, dtype=np.int32) + indices = [] + bl = np.zeros(N) + for i, n in enumerate(order): + ch = [idx[c] for c in sorted(g.successors(n), key=str)] + indices.extend(ch) + indptr[i + 1] = indptr[i] + len(ch) + for c in g.successors(n): + bl[idx[c]] = _length(g, n, c) + tree = csr_array((np.ones(len(indices), dtype=bool), np.array(indices, dtype=np.int32), indptr), shape=(N, N)) + return PhyloTree(tree=tree, names=np.array(order, dtype=object), branch_lengths=bl) + + +def load_tree(source, keep_leaves: Iterable[str] | None = None, collapse_unifurcations: bool = True) -> PhyloTree: + """Load a tree from a Newick path/string, a networkx DiGraph, or a CassiopeiaTree. + + :param keep_leaves: If given, prune to these leaves (e.g. those with expression data). + """ + if isinstance(source, nx.DiGraph): + g = source + elif hasattr(source, "get_branch_length") and hasattr(source, "edges"): + g = cassiopeia_to_graph(source) + else: + g = newick_to_graph(source) + if keep_leaves is not None or collapse_unifurcations: + if keep_leaves is None: + keep_leaves = [n for n in g if g.out_degree(n) == 0] + g = prune_graph(g, keep_leaves, collapse_unifurcations) + return graph_to_phylotree(g) + + +def lca_character_states(pt: PhyloTree, leaf_characters: np.ndarray, missing: int = -1) -> np.ndarray: + """Cassiopeia-style ancestral reconstruction: a node keeps a character state only if all children + that observe it agree; otherwise it is uncut (0). Missing data propagates only if all children miss it. + + :param leaf_characters: (N by C) integer matrix; rows for internal nodes are ignored. + """ + chars = np.array(leaf_characters, dtype=int, copy=True) + for p in np.nonzero(~pt.is_leaf)[0][::-1]: + ch = pt.tree.indices[pt.tree.indptr[p] : pt.tree.indptr[p + 1]] + sub = chars[ch] + state = np.zeros(sub.shape[1], dtype=int) + for c in range(sub.shape[1]): + vals = sub[:, c][sub[:, c] != missing] + if vals.size == 0: + state[c] = missing + elif np.all(vals == vals[0]): + state[c] = vals[0] + chars[p] = state + return chars + + +def mutation_branch_lengths(pt: PhyloTree, node_characters: np.ndarray, missing: int = -1) -> np.ndarray: + """Number of characters that change along the edge into each node (0 for the root).""" + par = pt.parents + bl = np.zeros(len(pt.names)) + c = node_characters[1:] + p = node_characters[par[1:]] + present = (c != missing) & (p != missing) + bl[1:] = np.sum(present & (c != p), axis=1) + return bl + + +def align_observations(pt: PhyloTree, obs_names: Sequence[str] | np.ndarray, obs: np.ndarray) -> np.ndarray: + """Place per-cell observations at the matching leaves; internal nodes and unmatched leaves get NaN.""" + obs = np.asarray(obs, dtype=float) + lookup = {str(n): i for i, n in enumerate(obs_names)} + out = np.full((len(pt.names), obs.shape[1]), np.nan) + for i, name in enumerate(pt.names): + if pt.is_leaf[i] and str(name) in lookup: + out[i] = obs[lookup[str(name)]] + return out + + +def to_lineage( + pt: PhyloTree, + obs_names: Sequence[str] | np.ndarray, + obs: np.ndarray, + E: Sequence, + branch_lengths: np.ndarray | None = None, +) -> LineageTree: + """Build a LineageTree with NaN observations at internal nodes and the tree's branch lengths.""" + bl = pt.branch_lengths if branch_lengths is None else branch_lengths + lin = LineageTree(pt.tree, E, obs=align_observations(pt, obs_names, obs), branch_lengths=np.asarray(bl, float)) + lin.names = pt.names + return lin