PEF Framework Reproduction

Reduced-scale experimental reproduction of a privacy-enhancing framework (PEF) across three modalities — clinical text, face images, and ECG signals — plus a canary/honeytoken monitoring-layer test. Every result is produced by running real code against real public data (no fabricated numbers).

Layout

data/                      raw input datasets only (nothing computed lives here)
experiment/                pipeline code that applies PEF's protection and measures
                           privacy-attack success + downstream utility, per modality
results/                   the three headline figures + the code that generates them
medical_AI_security_v1/    private submodule, see note below
requirements.txt

experiment/ and results/ contain only code — no precomputed outputs are checked in. Running the pipelines regenerates everything (metrics, intermediate artifacts, and the three figures) from the raw data in data/. That's what makes this reproducible rather than a static snapshot.

About the medical_AI_security_v1 submodule: this links to an earlier, version-1 iteration of this experiment. It is kept for historical/internal reference only, which is why it stays in a private repo.

Datasets

• Text: mtsamples.csv — public medical transcription samples.
• Image: 6,000 face photos + attribute/identity/landmark manifest, sourced from the Hugging Face mirror negedng/CelebA-attrs-identity of the CelebA dataset (Liu et al., 2015).
• Signal: PTB-XL v1.0.3 (3 of 22 records100 batches, 2,977 records; Wagner et al., 2020) and the full ECG-ID database (90 subjects, 310 records; Lugovaya, 2005), both from PhysioNet (Goldberger et al., 2000).

Setup

Requires Python 3.13 (tested on 3.13.14) and about 2GB of disk for the virtual environment + model/dataset caches (on top of the ~200MB of raw data already in this repo).

cd medical_AI_security_v2       # repo root
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install --no-deps facenet-pytorch==2.6.0

The facenet-pytorch install is deliberately separate and uses --no-deps: its own pinned dependencies (numpy<2.0, torch<2.3, Pillow<10.3) are stale and conflict with the newer numpy/torch/Pillow versions everything else in this repo uses, even though the library itself runs fine against them. spaCy's en_core_web_lg model (needed by Presidio for text PHI detection) is already pulled in directly by requirements.txt as a wheel URL, so no separate spacy download step is needed.

Then, from the repo root, before running any experiment script:

source experiment/shared/env.sh

This activates the venv if present, points all caches (pip/HuggingFace/torch/ matplotlib) at hidden folders inside the repo instead of your home directory, and adds experiment/shared to PYTHONPATH so the modality scripts can import evaluate. Always run this from the repo root — it uses your current directory to find everything.

Running the experiments

Each modality's scripts write their outputs into a fresh top-level folder at the repo root named after that modality (e.g. text/, image/, signal/, monitoring/) — separate from experiment/<modality>/, where the code lives. These output folders are not checked into the repo; running the pipelines creates them.

Text (run in order from the repo root):

python experiment/text/01_prepare_subset.py
python experiment/text/02_inject_phi.py
python experiment/text/03_component1_extraction.py
python experiment/text/04_presidio_detect.py
python experiment/text/05_phi_metrics.py
python experiment/text/06_masking_and_redaction.py
python experiment/text/07_pef_and_embeddings.py
python experiment/text/08_utility_eval.py
python experiment/text/09_tradeoff_sweep.py
python experiment/text/10_assemble_results.py   # writes text/results.json

Image (run in order from the repo root — note these read raw photos directly from data/image/, not from a live Hugging Face stream, so this runs fully offline against the committed data):

python experiment/image/scripts/01_build_dataset.py
python experiment/image/scripts/02_make_splits.py
python experiment/image/scripts/03_build_protected.py
python experiment/image/scripts/04_privacy_attack.py
python experiment/image/scripts/05_utility_task.py
python experiment/image/scripts/06_write_results.py   # writes image/results.json
python experiment/image/scripts/07_secondary_rosy_cheeks.py   # optional secondary metric

Signal (single script, reads data/signal/):

python experiment/signal/run_experiment.py   # writes signal/results.json + signal/tradeoff.json

Monitoring (single script, reads data/text/mtsamples.csv directly — independent of the text pipeline above):

python experiment/monitoring/run_experiment.py   # writes monitoring/results.json

Generating the figures

Once the relevant results.json/tradeoff.json files exist (see above), run from the repo root (these three are standalone — no env.sh needed, just matplotlib):

python results/figure_1_pef_before_after.py   # needs text/, image/, signal/ results.json
python results/figure_2_tradeoff.py           # needs text/ and signal/ tradeoff.json
python results/figure_3_monitoring.py         # needs monitoring/ results.json

Each overwrites its corresponding PNG in results/ in place.