Hi Authors,
First of all, thank you for your excellent work! The EEG-Conformer is a highly inspiring architecture and a great contribution to the EEG emotion recognition community.
While studying your code, specifically in conformer_seed_1s_5fold.py, I noticed a potential Data Leakage issue caused by Within-Trial Splitting in the get_source_data function.
Description of the Issue:
In the current implementation of get_source_data, the 5-fold splitting logic divides each individual trial into 5 parts (4 parts for training, 1 part for testing):
Python
one_fold_num = np.shape(tmp_tri)[0] // 5
...
test_idx = np.arange(one_fold_num * fold, one_fold_num * (fold+1))
train_idx = np.delete(tmp_tri_idx, test_idx)
self.train_data.append(tmp_tri[train_idx])
self.test_data.append(tmp_tri[test_idx])
Because EEG signals are highly continuous time-series data, adjacent segments within the same trial (watching the same movie clip) share nearly identical background noise, baseline drift, and muscle artifacts.
By putting 80% of Trial A in the training set and 20% of Trial A in the test set, the model can achieve extremely high accuracy by simply "memorizing" the temporal correlation or baseline state of that specific trial, rather than learning generalizable emotion-related EEG features.
Proposed Solution:
To prevent temporal data leakage, the cross-validation should be performed at the Trial level (Leave-Trials-Out) rather than the Segment level.
For the SEED dataset (which has 15 trials per session corresponding to 3 emotions), a rigorous 5-fold CV could use 12 complete trials for training and 3 completely unseen trials for testing in each fold (ideally ensuring class balance in the test set).
I wanted to bring this to your attention as it might affect the reproducibility of the true generalization performance of the model. I would love to hear your thoughts on this!
Thanks again for releasing this awesome codebase!
Best regards.
Hi Authors,
First of all, thank you for your excellent work! The EEG-Conformer is a highly inspiring architecture and a great contribution to the EEG emotion recognition community.
While studying your code, specifically in conformer_seed_1s_5fold.py, I noticed a potential Data Leakage issue caused by Within-Trial Splitting in the get_source_data function.
Description of the Issue:
In the current implementation of get_source_data, the 5-fold splitting logic divides each individual trial into 5 parts (4 parts for training, 1 part for testing):
Python
one_fold_num = np.shape(tmp_tri)[0] // 5
...
test_idx = np.arange(one_fold_num * fold, one_fold_num * (fold+1))
train_idx = np.delete(tmp_tri_idx, test_idx)
self.train_data.append(tmp_tri[train_idx])
self.test_data.append(tmp_tri[test_idx])
Because EEG signals are highly continuous time-series data, adjacent segments within the same trial (watching the same movie clip) share nearly identical background noise, baseline drift, and muscle artifacts.
By putting 80% of Trial A in the training set and 20% of Trial A in the test set, the model can achieve extremely high accuracy by simply "memorizing" the temporal correlation or baseline state of that specific trial, rather than learning generalizable emotion-related EEG features.
Proposed Solution:
To prevent temporal data leakage, the cross-validation should be performed at the Trial level (Leave-Trials-Out) rather than the Segment level.
For the SEED dataset (which has 15 trials per session corresponding to 3 emotions), a rigorous 5-fold CV could use 12 complete trials for training and 3 completely unseen trials for testing in each fold (ideally ensuring class balance in the test set).
I wanted to bring this to your attention as it might affect the reproducibility of the true generalization performance of the model. I would love to hear your thoughts on this!
Thanks again for releasing this awesome codebase!
Best regards.