Repository navigation
Recognize Empatica E4 exports in dataset validation - #186
Merged
Merged
Conversation
HR.csv shares the start-time and sample-rate layout of the other sensors. IBI.csv starts with the start time and "IBI", then has offset and interval columns; an IBI.csv with no beats is reported as information. E4 file names are now matched without regard to letter case, and tags.csv looks for neighboring sensors by listing its folder so this also works on case-sensitive file systems. Privacy warnings go back to one per column. Fields beyond a row's expected columns are still scanned and reported under "extra field".
A column holding both an email address and a phone number previously got one warning naming whichever pattern matched first. Each matching pattern is now reported, so some datasets will see more privacy warnings.
Integrity, quality, and privacy checks found files with a case-sensitive "*.csv" glob, so files such as EDA.CSV were never checked. CSV and text files are now found by extension in any letter case. The E4 tags timestamp exemption applied to every field in the row, which hid phone numbers in surplus fields. It now applies only to the layout's named timestamp columns. A recognized IBI.csv now confirms a neighboring tags.csv, like the sensor files do.
The IBI test only asserted that valid data gave no warnings, which would also hold if quality checks skipped every IBI row. Add an out-of-range interval and assert it is reported under the ibi column.
E4 is a niche format, so the README now has a short note on what the validator does with it. The file layouts, column names, and tags/IBI rules move to the empatica module docstring.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
csv.Snifferwas interpreting digits in headerless decimal sensor files as delimiters, causing false row-width errors. Files that relied on space or colon as a delimiter will now be parsed as one-column CSV.IBI.csvby its start-time andIBIfirst row, and check its rows astime_offsetandibi. AnIBI.csvwith no detected beats is reported as information.tags.csvfinds neighboring sensor or IBI files by listing its folder, so this also works on case-sensitive file systems.EDA.CSVorNOTES.TXTwere previously skipped.tags.csvin a matching E4 session as "no event markers" (informational), not as a broken CSV or an empty-file warning.This is limited to the legacy E4 Connect layout documented by Empatica: https://www.empatica.com/blog/decoding-wearable-sensor-signals-what-to-expect-from-your-e4-data/. It does not establish that timestamps have been de-identified.
Verification
pytest -q— 195 passed, 1 skippedgit diff --check— passed