Skip to content

PHASE1_VALIDATION_RESULTS.md

Latest commit

 

History

History
75 lines (57 loc) · 4.15 KB

PHASE1_VALIDATION_RESULTS.md

File metadata and controls

75 lines (57 loc) · 4.15 KB
Raw

Phase 1 Validation on Existing Benchmarks (Months 1–6)

Date: 2026-06-30

Scope executed

  • Benchmark script: benchmarks/local_csv_benchmark.py
  • Dataset: local data/mimic-iv-demo CSV files
  • Grid: subsets QUARTER, HALF, FULL × configs RHT-TinyEmbed, RHT-Full
  • Epochs: 1 (pilot) and 5 (publication pass)
  • Seeds: 0, 1, 2, 3, 4 (5 runs)

Commands executed

  • Single-grid run (sanity):
    • /home/lansiauxedouard/Desktop/multitable-5D-analysis-main/.venv/bin/python benchmarks/local_csv_benchmark.py --subsets QUARTER HALF FULL --configs RHT-TinyEmbed RHT-Full --epochs 1 --out-dir benchmark_results
  • Multi-seed run:
    • for s in 0 1 2 3 4; do /home/lansiauxedouard/Desktop/multitable-5D-analysis-main/.venv/bin/python benchmarks/local_csv_benchmark.py --subsets QUARTER HALF FULL --configs RHT-TinyEmbed RHT-Full --epochs 1 --seed $s --out-dir benchmark_results; done

Raw result files used for aggregation

  • benchmark_results/local_csv_benchmark_20260630_164343.csv
  • benchmark_results/local_csv_benchmark_20260630_164427.csv
  • benchmark_results/local_csv_benchmark_20260630_164512.csv
  • benchmark_results/local_csv_benchmark_20260630_164556.csv
  • benchmark_results/local_csv_benchmark_20260630_164640.csv

Publication pass (5 epochs)

  • benchmark_results/local_csv_benchmark_20260630_165240.csv
  • benchmark_results/local_csv_benchmark_20260630_165335.csv
  • benchmark_results/local_csv_benchmark_20260630_165431.csv
  • benchmark_results/local_csv_benchmark_20260630_165527.csv
  • benchmark_results/local_csv_benchmark_20260630_165623.csv

Aggregated results (mean ± std over 5 seeds)

Subset Config Throughput (rows/s) Compression ratio Train time (s) Peak memory (MB)
QUARTER RHT-TinyEmbed 27813.74 ± 6893.89 1.53 ± 0.35 7.96 ± 0.05 64.03 ± 0.05
QUARTER RHT-Full 35688.79 ± 8994.12 1.53 ± 0.35 6.21 ± 0.06 3.57 ± 0.04
HALF RHT-TinyEmbed 59157.51 ± 6955.55 2.22 ± 0.26 5.84 ± 0.03 3.53 ± 0.01
HALF RHT-Full 55303.44 ± 6439.80 2.22 ± 0.26 6.25 ± 0.04 3.62 ± 0.01
FULL RHT-TinyEmbed 119481.05 ± 958.89 4.09 ± 0.00 6.42 ± 0.05 5.55 ± 0.00
FULL RHT-Full 111047.90 ± 575.92 4.09 ± 0.00 6.91 ± 0.04 5.54 ± 0.00

Key outcomes

  1. Hypergraph construction stability: all runs complete on the full QUARTER/HALF/FULL grid (no VS Code crash).
  2. Scalability trend: throughput scales strongly with subset size, peaking at ~120k rows/s.
  3. Compression trend: compression ratio increases from ~1.5 (QUARTER) to ~4.1 (FULL).
  4. Config behavior:
    • RHT-TinyEmbed is faster on FULL subset.
    • RHT-Full has lower peak memory on QUARTER in this setup.
  5. Current limitations for D3/D4/D5 in this quick benchmark:
    • Rare-category recall and relation-discovery F1 are near 0 in this short-run setting.
    • Temporal imputation is unstable in demo-scale runs and needs dedicated tuning.

Publication pass summary (5 epochs, mean ± std over 5 seeds)

Subset Config Throughput (rows/s) Compression ratio Train time (s) Peak memory (MB)
QUARTER RHT-TinyEmbed 25193.66 ± 6252.33 1.53 ± 0.35 8.79 ± 0.10 64.02 ± 0.04
QUARTER RHT-Full 25791.34 ± 6739.89 1.53 ± 0.35 8.61 ± 0.21 3.62 ± 0.04
HALF RHT-TinyEmbed 50818.80 ± 6176.44 2.22 ± 0.26 6.81 ± 0.05 3.56 ± 0.01
HALF RHT-Full 38137.71 ± 4545.35 2.22 ± 0.26 9.07 ± 0.07 3.67 ± 0.01
FULL RHT-TinyEmbed 101011.85 ± 654.73 4.09 ± 0.00 7.59 ± 0.05 5.55 ± 0.00
FULL RHT-Full 74918.17 ± 410.67 4.09 ± 0.00 10.24 ± 0.06 5.54 ± 0.00

Interpretation: the publication pass keeps the same scalability/compression trend, with slower wall-clock speed than the 1-epoch pilot (expected) and improved stability of runtime estimates across seeds.

Notes

  • These results are intended as Phase 1 benchmark validation outputs for manuscript update.
  • For publication-grade D3/D4/D5 comparisons, use longer training and full task protocol (experiments/run.py on Synthea and/or full MIMIC-IV access when available).