Skip to content

Recognize Empatica E4 exports in dataset validation - #186

Merged
tompollard merged 10 commits into
mainfrom
fix/empatica-validation
Oct 20, 2026
Merged

tompollard merged 10 commits into
mainfrom
fix/empatica-validation

Conversation

@tompollard

@tompollard tompollard commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Restrict CSV delimiter detection to comma, semicolon, tab, or pipe for all CSV files. csv.Sniffer was interpreting digits in headerless decimal sensor files as delimiters, causing false row-width errors. Files that relied on space or colon as a delimiter will now be parsed as one-column CSV.
  • Recognize Empatica E4 Connect sensor files (ACC, BVP, EDA, HR, TEMP) by their timestamp and sample-rate rows. Avoid treating those rows as column headers.
  • Recognize E4 IBI.csv by its start-time and IBI first row, and check its rows as time_offset and ibi. An IBI.csv with no detected beats is reported as information.
  • Match E4 file names in any letter case. tags.csv finds neighboring sensor or IBI files by listing its folder, so this also works on case-sensitive file systems.
  • Find CSV and text files by extension in any letter case in the integrity, quality, and privacy checks. Files such as EDA.CSV or NOTES.TXT were previously skipped.
  • Treat an empty tags.csv in a matching E4 session as "no event markers" (informational), not as a broken CSV or an empty-file warning.
  • Scan headerless E4 tags as data. Do not label nine- or ten-digit Unix event timestamps as phone numbers, including timestamps shifted to dates before 2000 or after 2100; the exemption applies only to the event-time column, so phone-like values in surplus fields and other values are still reported.
  • Check E4 sensor data by position under distinct channel names, skipping timestamp and sample-rate rows.
  • Scan surplus fields in malformed CSV rows for private information under an "extra field" column, without aborting privacy checks.
  • Report each privacy pattern found in a column, rather than only the first one matched. A column holding both an email address and a phone number now gets two warnings, so privacy warning counts can increase for any CSV dataset.
  • Document the behavior and add regression tests for the reported cases and ordinary CSV safeguards.
  • Add a small synthetic E4 export as a documented example and test the complete Python and CLI validation workflow against it.
  • Resolve CSV layouts once for all validator checks. The E4 detector supplies its layout, while ordinary headered CSV remains the default. A new format can add a detector without changing each check.

This is limited to the legacy E4 Connect layout documented by Empatica: https://www.empatica.com/blog/decoding-wearable-sensor-signals-what-to-expect-from-your-e4-data/. It does not establish that timestamps have been de-identified.

Verification

  • pytest -q — 195 passed, 1 skipped
  • git diff --check — passed

HR.csv shares the start-time and sample-rate layout of the other sensors.
IBI.csv starts with the start time and "IBI", then has offset and interval
columns; an IBI.csv with no beats is reported as information.

E4 file names are now matched without regard to letter case, and tags.csv
looks for neighboring sensors by listing its folder so this also works on
case-sensitive file systems.

Privacy warnings go back to one per column. Fields beyond a row's expected
columns are still scanned and reported under "extra field".
A column holding both an email address and a phone number previously got
one warning naming whichever pattern matched first. Each matching pattern
is now reported, so some datasets will see more privacy warnings.
Integrity, quality, and privacy checks found files with a case-sensitive
"*.csv" glob, so files such as EDA.CSV were never checked. CSV and text
files are now found by extension in any letter case.

The E4 tags timestamp exemption applied to every field in the row, which
hid phone numbers in surplus fields. It now applies only to the layout's
named timestamp columns.

A recognized IBI.csv now confirms a neighboring tags.csv, like the sensor
files do.
The IBI test only asserted that valid data gave no warnings, which would
also hold if quality checks skipped every IBI row. Add an out-of-range
interval and assert it is reported under the ibi column.
E4 is a niche format, so the README now has a short note on what the
validator does with it. The file layouts, column names, and tags/IBI rules
move to the empatica module docstring.
@tompollard
tompollard merged commit 8b389c7 into main Oct 20, 2026
2 checks passed
@tompollard
tompollard deleted the fix/empatica-validation branch October 20, 2026 13:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant