feat(integration): port Valentin Laborie's ETH lifecycle GMM-HMM to hmm-studio idioms

Reproduces the 2025 S2 deliverable (Modèles probabilistes/gmmhmm Full.py)
end-to-end through hmm-studio's declarative APIs. Private data is referenced
via env var HMM_VALENTIN_ETH_PATH and is never committed.

Engine additions:
- prep ops log1p (skip-negative-by-default) and drop_low_variance
  (encodes the std-threshold + zeros-fraction filter pattern). 5 unit tests.
- bundled recipe valentin_eth: 365-day rolling mean -> dropna ->
  drop_low_variance -> log1p -> zscore. End-to-end produces 3402 rows of
  standardised features from the raw 3766-row CSV.

Examples + notebook:
- examples/valentin_eth_3regime_gmm.yaml: 3-state GMM topology (diag,
  n_mix=3, ergodic). The YAML documents Valentin's strict left-right
  original and explains why ergodic is used here (hmm-studio re-fits
  startprob via M-step, which NaNs under strict left-right; Valentin's
  hmmlearn params=mcw freeze isn't exposed).
- notebooks/10_valentin_eth_gmm_hmm.ipynb: 15-cell narrative wiring CSV
  load -> recipe -> PCA(n=2) -> fit -> diagnostics, with a comparison
  table of what was ported faithfully vs adapted to studio idioms.

Regression:
- tests/test_valentin_eth_regression.py: 6 tests pinning log-likelihood
  (-636 +/- 5%), BIC, PCA explained variance, EM convergence, prep-output
  shape, and phase populations. Auto-skipped when env var unset.

Safety:
- .gitignore: *JDD_ETH*.csv, *JDD_BTC*.csv, **/private_data/,
  notebooks/data/ patterns added to prevent private CSV commits.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Author: RoJLD

.gitignore

@@ -40,3 +40,11 @@ src/hmm_studio/frontend/dist/
 # Frontend build copied into the server package (built artifact, regenerated)
 src/hmm_studio/server/static/index.html
 src/hmm_studio/server/static/assets/
+
+# Private datasets — referenced via env vars from notebooks / regression
+# tests, must NEVER end up in commits. Add new patterns here whenever a
+# private CSV/parquet is wired into a notebook (env-var path pattern).
+*JDD_ETH*.csv
+*JDD_BTC*.csv
+**/private_data/
+notebooks/data/

CHANGELOG.md

@@ -4,6 +4,50 @@ All notable changes to `hmm-studio` are documented here. This project follows
 [Semantic Versioning](https://semver.org/) and the
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) format.
 
+## [Unreleased]
+
+### Added — engine (`hmm_core.prep`)
+
+- Two new prep ops:
+  - **`log1p`** — apply `np.log1p` to listed (or all numeric) columns; by
+    default skips columns containing any negative value so it chains
+    safely after generic preprocessing.
+  - **`drop_low_variance`** — drop columns that are near-constant
+    (`std < threshold`) or mostly-zero (`zeros_fraction > threshold`).
+    Encodes the "degenerate column" filter pattern that came up in the
+    Valentin ETH port.
+- New bundled recipe `valentin_eth` (5-step pipeline: 365-day rolling
+  mean → dropna → drop_low_variance → log1p → zscore) ready for use via
+  `Pipeline.from_recipe("valentin_eth")`.
+
+### Added — examples + notebooks
+
+- `examples/valentin_eth_3regime_gmm.yaml`: 3-state GMM topology (diag
+  covariance, `n_mix=3`, ergodic; documents Valentin's strict left-right
+  original and why it would need a frozen-startprob M-step to work
+  as-is).
+- `notebooks/10_valentin_eth_gmm_hmm.ipynb`: ports Valentin Laborie's
+  *2025 S2* ETH lifecycle GMM-HMM end-to-end through hmm-studio idioms
+  (private data: set `HMM_VALENTIN_ETH_PATH` env var to the CSV path
+  before running).
+
+### Added — tests
+
+- `tests/test_valentin_eth_regression.py`: 6 regression tests pinning the
+  log-likelihood (−636 ± 5%), BIC, PCA explained variance, EM convergence,
+  prep-output shape, and phase populations. **Auto-skipped** when
+  `HMM_VALENTIN_ETH_PATH` is unset (private dataset not in the repo).
+- 5 new prep-op tests covering `log1p` (explicit columns, negative-skip,
+  forced-on-negative) and `drop_low_variance` (default + thresholds).
+
+### Changed — `.gitignore`
+
+- Added `*JDD_ETH*.csv`, `*JDD_BTC*.csv`, `**/private_data/`,
+  `notebooks/data/` patterns to prevent accidental commit of private
+  research datasets referenced from notebooks via env vars.
+
+---
+
 ## [1.1.0] — 2026-05-23
 
 Distribution-surface release: `hmm-studio` becomes pip-installable and

examples/valentin_eth_3regime_gmm.yaml

@@ -0,0 +1,44 @@
+name: valentin_eth_3regime_gmm
+n_states: 3
+state_names: [accumulation, expansion, distribution]
+
+# GMM emission : each state hosts a 3-component Gaussian mixture on the
+# 2-D PCA space of the preprocessed ETH on-chain indicators (see
+# notebooks/10_valentin_eth_gmm_hmm.ipynb for the upstream pipeline).
+#
+# Valentin's original used covariance_type='full' with a hand-crafted
+# init from 3 equal-length data segments. hmm-studio's kmeans init for
+# GMM emits diag-shaped covars, so we use diag here for an out-of-the-box
+# fit. In 2-D PCA space the practical difference is small; the notebook
+# documents the original full-covariance recipe for reference.
+emission:
+  type: gmm
+  covariance_type: diag
+  n_features: 2
+  n_mix: 3
+
+# Ergodic topology (no allowed_transitions block = all transitions allowed).
+#
+# Valentin's original was strict left-right (Bakis) with a hand-clamped
+# startprob = [1, 0, 0] and ``params='mcw'`` to freeze startprob/transmat
+# in hmmlearn's M-step. hmm-studio always re-fits startprob from
+# posteriors, which together with strict left-right produces a 0/0 NaN
+# in the M-step (state 0 is only visited at t=0; its responsibility sum
+# collapses).
+#
+# Workaround : leave the topology ergodic. The data is sufficiently
+# regime-like that EM naturally discovers a quasi-left-right structure
+# (self-loop probabilities ~ 0.998, cross-transitions ~ 0.001-0.002),
+# which is the practical answer Valentin's hard constraint was trying
+# to encode. The notebook compares the two formulations.
+
+startprob: uniform
+
+init:
+  strategy: kmeans
+  seed: 42
+
+fit:
+  algorithm: baum_welch
+  n_iter: 500
+  tol: 1.0e-4

notebooks/10_valentin_eth_gmm_hmm.ipynb

@@ -0,0 +1,229 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Valentin's ETH lifecycle GMM-HMM — ported to hmm-studio\n",
+    "\n",
+    "Reproduces Valentin Laborie's *2025 S2* GMM-HMM on Ethereum on-chain indicators, using only `hmm-studio` idioms (declarative `Topology`, bundled `prep` recipe, `fit_gmm_nhmm`-free path through the standard `fit()` since the model is homogeneous).\n",
+    "\n",
+    "**Original deliverable** : `Experiment.Crypto.2025S2.LifeCycle/Modèles probabilistes (py)/gmmhmm Full.py`\n",
+    "\n",
+    "**Data** : `JDD_ETH_après la corrélation_FINAL_HMM.csv` (~3766 daily rows, 4 features: `AdrActCnt`, `ROI30d`, `TxCnt`, `SplyExNtv`). The file is **private** and intentionally not committed to the repo. Point the env var `HMM_VALENTIN_ETH_PATH` at your local copy."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Load the private CSV\n",
+    "\n",
+    "Set `HMM_VALENTIN_ETH_PATH` to the absolute path of `JDD_ETH_après la corrélation_FINAL_HMM.csv` before launching the notebook."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import pandas as pd\n",
+    "\n",
+    "csv_path = os.environ.get(\"HMM_VALENTIN_ETH_PATH\")\n",
+    "if not csv_path:\n",
+    "    raise SystemExit(\n",
+    "        \"Set HMM_VALENTIN_ETH_PATH=<...>/JDD_ETH_après la corrélation_FINAL_HMM.csv \"\n",
+    "        \"before running this notebook.\"\n",
+    "    )\n",
+    "\n",
+    "df = (\n",
+    "    pd.read_csv(csv_path, encoding=\"ISO-8859-1\", parse_dates=[\"date\"])\n",
+    "    .sort_values(\"date\")\n",
+    "    .set_index(\"date\")\n",
+    ")\n",
+    "df.shape, df.columns.tolist()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Apply Valentin's preprocessing as an hmm-studio recipe\n",
+    "\n",
+    "The recipe `valentin_eth` chains the 5 preprocessing steps from the original script :\n",
+    "\n",
+    "1. 365-day rolling mean on every input feature (de-noise)\n",
+    "2. dropna (drop the first 364 boundary rows)\n",
+    "3. drop_low_variance (Valentin's `zeros_frac > 0.5 or std < 1e-8` filter)\n",
+    "4. log1p on every non-negative column\n",
+    "5. zscore on the surviving features\n",
+    "\n",
+    "PCA is intentionally NOT inside the recipe (the prep layer is pandas-only by design)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from hmm_core.prep import Pipeline\n",
+    "\n",
+    "prep = Pipeline.from_recipe(\"valentin_eth\")\n",
+    "result = prep.fit_transform(df)\n",
+    "result.df.shape, result.df.columns.tolist()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Sanity : every feature is now centered + scaled\n",
+    "result.df.describe().T[[\"mean\", \"std\", \"min\", \"max\"]]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. PCA to 2 components\n",
+    "\n",
+    "Valentin uses PCA to reduce 4 standardized features → 2 dimensions, then fits the GMM-HMM on the 2-D scores. We do the same here using `sklearn.decomposition.PCA` outside the prep recipe."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sklearn.decomposition import PCA\n",
+    "\n",
+    "pca = PCA(n_components=2, random_state=42)\n",
+    "X_pca = pca.fit_transform(result.df.values)\n",
+    "dates = result.df.index\n",
+    "print(f\"X_pca shape : {X_pca.shape}\")\n",
+    "print(f\"Explained variance ratio : {pca.explained_variance_ratio_}\")\n",
+    "print(f\"Cumulative : {pca.explained_variance_ratio_.sum():.3f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Declare the topology and fit\n",
+    "\n",
+    "`examples/valentin_eth_3regime_gmm.yaml` carries the topology : 3 states (accumulation / expansion / distribution), GMM emissions with `n_mix=3`, diagonal covariances, ergodic transitions (Valentin's original strict left-right is documented in the YAML — we use ergodic here because hmm-studio's M-step would NaN under strict left-right + GMM, and the data is regime-like enough that EM naturally discovers a quasi-left-right structure)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from hmm_core.io import load_topology\n",
+    "from hmm_core.fit import fit\n",
+    "\n",
+    "topo = load_topology(\"../examples/valentin_eth_3regime_gmm.yaml\")\n",
+    "fitted = fit(topo, X_pca, seed=42)\n",
+    "fitted   # rich HTML view : stats + transmat heatmap"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f\"Log-likelihood : {fitted.log_likelihood:.2f}\")\n",
+    "print(f\"Per observation : {fitted.log_likelihood / len(X_pca):.4f}\")\n",
+    "print(f\"BIC : {fitted.bic:.2f}\")\n",
+    "print(f\"AIC : {fitted.aic:.2f}\")\n",
+    "print(f\"EM iterations  : {fitted.n_iter_actual}\")\n",
+    "print(f\"Converged      : {fitted.converged}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Decode the lifecycle phases\n",
+    "\n",
+    "Viterbi on the fitted model gives the most likely sequence of latent states. The transition matrix typically shows three sticky regimes (self-loops near 0.998) with small cross-transitions — the natural lifecycle structure Valentin's strict left-right constraint was trying to encode."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "\n",
+    "states = fitted.model.predict(X_pca)\n",
+    "print(\"Transition matrix (rows = from, cols = to) :\")\n",
+    "print(fitted.model.transmat_.round(3))\n",
+    "print(\"\\nPhase frequencies :\")\n",
+    "for i, name in enumerate(topo.state_names):\n",
+    "    pct = (states == i).mean() * 100\n",
+    "    print(f\"  {i}  {name:15s} : {pct:5.1f} % of timeline\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Compare with the original Valentin script\n",
+    "\n",
+    "The full reference values you should see (PCA seed=42, hmm-studio kmeans init seed=42, hmmlearn deterministic EM) :\n",
+    "\n",
+    "| Metric | Reference (hmm-studio) |\n",
+    "|---|---|\n",
+    "| Log-likelihood total | ~ −636 |\n",
+    "| Log-likelihood / obs | ~ −0.187 |\n",
+    "| BIC | ~ 1678 |\n",
+    "| AIC | ~ 1371 |\n",
+    "| Converged | True |\n",
+    "| EM iterations | ~ 138 |\n",
+    "| PCA explained variance (2 PCs) | ~ 0.880 |\n",
+    "\n",
+    "These reference values are checked by `tests/test_valentin_eth_regression.py` (skipped if `HMM_VALENTIN_ETH_PATH` is unset, run in CI by exporting the path to a secret-mounted copy of the dataset)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## What was ported and what was kept original\n",
+    "\n",
+    "| Component | Valentin original | hmm-studio port |\n",
+    "|---|---|---|\n",
+    "| Preprocessing | inline pandas + sklearn (~50 lines) | bundled recipe `valentin_eth` (declarative YAML) |\n",
+    "| PCA | inline `sklearn.PCA(n_components=2)` | inline `sklearn.PCA` (unchanged — prep layer stays pandas-only) |\n",
+    "| Topology | hardcoded transmat / startprob in Python | declarative `examples/valentin_eth_3regime_gmm.yaml` |\n",
+    "| GMM covariance | `full` | `diag` (hmm-studio kmeans init doesn't emit full covars yet) |\n",
+    "| Topology shape | strict left-right with frozen startprob (`params='mcw'`) | ergodic with uniform startprob (avoids M-step NaN; EM rediscovers quasi-left-right) |\n",
+    "| EM engine | direct `hmmlearn.GMMHMM` | `hmm_core.fit` via `HMMBackend` protocol |\n",
+    "| Visualisations | manual matplotlib (regime timeline, PCA biplot, posterior probas) | `fitted._repr_html_()` + Web UI Results page |"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}