Skip to content

MODELS.md

Latest commit

 

History

History
312 lines (237 loc) · 8.64 KB

MODELS.md

File metadata and controls

312 lines (237 loc) · 8.64 KB
Raw

Models

Complete documentation for training models, split strategies, cross-validation, and hyperparameter optimization.

Available models

Model Type Input Description Kappa
CNN-LSTM Seq DL Raw signals (sequences) CNN + multi-epoch BiLSTM (best) 0.721
CNN1D DL Raw signals Convolutional network with residual blocks 0.680
XGBoost ML Features Gradient boosting with LOSO (selected) 0.675±0.097
LSTM Bi+Attn DL Raw signals Bidirectional LSTM with attention 0.651
Random Forest ML Features Tree ensemble, robust and fast 0.635
LSTM Unidir DL Raw signals Unidirectional LSTM (real-time) 0.530
LSTM Bidir DL Raw signals Bidirectional LSTM 0.521

Recommendations:

  • Best performance: CNN-LSTM Seq (κ=0.721, multi-epoch)
  • Single-epoch DL: CNN1D (κ=0.680)
  • Interpretable ML + LOSO: XGBoost (κ=0.675±0.097 in LOSO-CV; tuning on a subject holdout)
  • Real-time inference: Unidirectional LSTM (κ=0.530)

Recommended workflow

Extract features once, train many models

# 1. Extract features (~20-30 minutes)
python -m src.extract_features \
	--manifest data/processed/manifest_trimmed_resamp200.csv \
	--output data/processed/features_resamp200.parquet \
	--format parquet

# 2. Train models quickly
python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type random_forest \
	--output-dir models

python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type xgboost \
	--output-dir models

Tip: Use python -m src.extract_features --help and python -m src.models --help for all options.

Training from the CLI

Random Forest

python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type random_forest \
	--n-estimators 200 \
	--max-depth 20 \
	--output-dir models

XGBoost

python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type xgboost \
	--n-estimators 300 \
	--max-depth 8 \
	--learning-rate 0.1 \
	--output-dir models

CNN1D (requires TensorFlow)

python -m src.models \
	--manifest data/processed/manifest_trimmed_spt.csv \
	--model-type cnn1d \
	--n-filters 64 \
	--epochs 50 \
	--output-dir models

LSTM (requires TensorFlow)

python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type lstm \
	--lstm-units 128 \
	--sequence-length 5 \
	--epochs 50 \
	--output-dir models

Note: Each model has specific options. Use python -m src.models --help to see the full list of configurable hyperparameters.

Programmatic usage

from src.models import run_training_pipeline

metrics = run_training_pipeline(
	manifest_path="data/processed/manifest_trimmed_resamp200.csv",
		model_type="random_forest",
		output_dir="models",
		test_size=0.2,
		n_estimators=200,
		max_depth=None,
)

Manual feature extraction

from src.features import extract_features_from_session

features_df = extract_features_from_session(
	psg_path="data/processed/sleep_trimmed_resamp200/psg/SC4001E_trimmed_raw.fif",
	hypnogram_path="data/processed/sleep_trimmed_resamp200/hypnograms/SC4001E_trimmed_annotations.csv",
		epoch_length=30.0,
		sfreq=100.0,
)

Split strategies

By subject (default, recommended)

All epochs from a subject are kept in the same split (train/val/test) to avoid leakage.

python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type random_forest \
	--test-size 0.2 \
	--val-size 0.2

Temporal split

The most recent sessions per subject go to test/val.

python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type random_forest \
	--temporal-split \
	--output-dir models_temporal

Note: Requires epoch_time_start or epoch_index in the features.

Cross-validation

Subject K-Fold (default)

python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type random_forest \
	--cross-validate \
	--cv-strategy subject-kfold \
	--cv-folds 5

Leave-One-Subject-Out (LOSO)

Each fold leaves one full subject out. Ideal to evaluate generalization to unseen subjects.

python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type random_forest \
	--cross-validate \
	--cv-strategy loso \
	--strict-class-coverage

Group temporal

Respects temporal order within each subject during CV.

python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type random_forest \
	--cross-validate \
	--cv-strategy group-temporal

Hyperparameter optimization

The pipeline automatically detects whether it is the first run and enables Bayesian optimization with Optuna.

Force optimization

python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--model-type random_forest \
	--optimize \
	--n-iter-optimize 50

Optimization parameters

Flag Default Description
--optimize auto Force optimization
--n-iter-optimize 50 Number of Optuna trials
--cv-folds-optimize 3 Folds for internal CV

Class coverage

The split ensures all classes (W, N1, N2, N3, REM) are present in each set.

# Fail if any class is missing during CV
python -m src.models \
	--features-file data/processed/features_resamp200.parquet \
	--cross-validate \
	--strict-class-coverage

Deep learning models

CNN-LSTM Seq considerations

  • Input: sequences of N consecutive epochs (batch, seq_length, n_samples, n_channels)
  • seq_length=11 by default (~5.5 minutes of temporal context)
  • CNN extracts per-epoch features, BiLSTM models temporal context
  • Per-channel normalization statistics are stored in the model
  • Best performance (Kappa 0.72) due to multi-epoch context

CNN1D considerations

  • Input: raw signals (n_epochs, n_channels, n_samples)
  • Per-channel normalization statistics are stored in the model
  • Data augmentation via GaussianNoise
  • Optional residual connections

LSTM considerations

  • Input: per-epoch raw signals (n_epochs, n_samples, n_channels) — single-epoch classification
  • Per-channel normalization statistics are stored in the model
  • Trained variants: unidirectional, bidirectional, bidirectional + attention
  • Note: No sequences of adjacent epochs are used; each epoch is classified independently

Normalization

DL models store normalization statistics from the training split:

  • CNN1D: channel_means_, channel_stds_
  • LSTM: scaler_

Important: evaluate_model fails if these are missing to prevent data leakage.

Evaluation metrics

Metric Description
Accuracy Overall accuracy
Cohen's Kappa Agreement beyond chance
Macro F1 Average F1 over classes
Weighted F1 F1 weighted by support
Confusion matrix Per-class errors

Training outputs

models/
├── random_forest_model.pkl          # Trained model
├── random_forest_feature_names.pkl  # Feature names
├── random_forest_metrics.json       # Evaluation metrics
├── xgboost_model.pkl
├── cnn1d_model.keras                # Keras model
├── cnn1d_model_custom_attrs.json    # Attributes (scaler, etc.)
└── lstm_model.keras

Recommendations

Scenario Recommendation
Small dataset (<15 subjects) Use LOSO CV
Medium dataset (15–50 subjects) Subject K-Fold with 5 folds
Large dataset (>50 subjects) Simple train/val/test
Evaluate generalization LOSO + strict-class-coverage
Raw signals CNN1D
Temporal context LSTM with sequence_length 5–10

Obtained results

See Comparative Analysis for full details.

Note

Reports under docs/reports/ are currently Spanish-only, but they are mostly plots/tables and should be easy to interpret.

Model Cohen's Kappa Macro F1 Accuracy
CNN-LSTM Seq 0.721 73.19% 80.04%
CNN1D 0.680 70.83% 76.86%
XGBoost (LOSO-CV) 0.675±0.097 67.96%±9.85% 77.15%±6.97%
LSTM Bi+Attn 0.651 68.07% 74.64%
Random Forest 0.635 69.50% 72.82%
LSTM Unidir 0.530 58.59% 66.17%

N1 is the hardest class (F1 ~40%) across all models, due to its transitional nature.

Next step

For GPU training on Kaggle, see Kaggle Notebooks.