Research code for a learned surrogate of ML utility in synthetic tabular-data generation.
Prior experiments do not establish that the learned ML-utility loss improves strong synthesizers. They reused holdouts, targeted arbitrary historical scores, and used unvalidated surrogate gradients. Cached utility labels must be regenerated before another comparison.
eval_ml_utility()now supports separate train/validation/final-test partitions, derives multiclass labels only from train/validation data, and two-way estimator training does not inspect its test set while fitting.- TVAE, TabDDPM, REaLTabFormer, and LCT-GAN studies propagate separate validation and final-test partitions when provided.
- MLU now directly maximizes its surrogate when no target is provided; its heuristic gradient penalty is disabled by default.
- The surrogate gradient remains unvalidated, and soft/internal guidance still differs from final decoded tables for several synthesizers.
- Estimator hyperparameter selection now scores the validation partition (
eval_val=True), so the final test partition stays reserved. Estimator hyperparameters chosen before this change were selected on the same data used to report them, and are not trustworthy. - Utility targets are standardized per source dataset and the estimator head is linear, so trained checkpoints under
models/predict in the old raw scale and must be retrained. - Every cached utility label was generated with CatBoost's bare defaults (
epochs=1), not the tuned hyperparameters inml_utility/params/, verified byte-exact against all four datasets (2026-08-08); regenerating labels must fix this, not just the 3-way split.
Treat MLU results as exploratory. Before making comparative claims, use independent generator/CatBoost-tuning/final-test splits, repeated paired seeds, held-out surrogate calibration, and a direct test that an MLU-proposed generator step beats an equal-norm random step on true held-out utility.
ml_utility_loss.loss_learning.evaluation exports optional helpers for these gates:
split_experiment_data()creates disjoint generator, CatBoost-selection, and final-test DataFrames.run_paired_benchmark()runs a caller-provided baseline/MLU synthesizer callback over matched seeds and returns per-seed deltas plus an approximate 95% interval.run_local_update_test()compares caller-provided equal-norm MLU and random updates on a held-out true-utility evaluator.
The callbacks intentionally own model construction, cloning, and sampling because each synthesizer has a different differentiable representation.
A dated failure diagnosis (why prior MLU runs helped only weak synthesizers — proxy leakage, an unsupervised surrogate gradient, and guided-tensor mismatches) and a ranked fix list with a per-synthesizer audit are in CLAUDE.md under "Diagnosis". A wrong-activation bug in the TVAE guided tensor is fixed (straight-through one-hot for categorical spans); LCT-GAN was already correct; tab_ddpm and REaLTabFormer need a larger redesign. The go/no-go gate is loss_learning/evaluation/gate_a_tvae.py:run_gate_a (built on run_local_update_test): an MLU decoder step must beat an equal-norm random step on true held-out utility, and passes only if the paired 95% CI is above zero.
A separate dated audit of training throughput, the data path, and the Optuna search is in CLAUDE.md under "Efficiency and tuning audit". It has now been worked through: per-epoch Optuna pruning, an in-memory sample cache, standardized targets under a linear head, a Spearman metric and an optional pairwise ranking loss for the rank-correlation gate, a single-objective objective_2, opt-in gradient metrics during evaluation, and the removal of dead search dimensions and per-batch clear_memory() calls. Studies must now supply their own pruner and use direction="minimize". What is left: the per-batch device syncs, which are the one remaining throughput item, and then deliberate omissions — the row-index cache key (in-memory caching turned it into a memory cost rather than a throughput one), AMP and TF32, conditional SDPA, and the caller-side items (ASHA, multi-seed re-runs). None of it is measured yet — the changes are syntax-checked only.
A literature review dated 2026-07-31 is in CLAUDE.md under "Literature review", written with the criticism of each result recorded next to it. Nothing from it is implemented. The findings that bear on the open gates: a scalar-quality predictor trained with per-sample MSE against a drifting target collapses to the mean, and the published fix is a ranking objective rather than target rescaling, which is the failure this project has been patching with std and mean-prediction penalties; the progressive dataset sizing in SizeScheduler is the part of curriculum learning that survives replication, while example-difficulty ordering is not; a cheap random-forest baseline on engineered meta-features should be run before any further architecture work; and a tabular prior-fitted network would make regenerating the utility labels far cheaper, though its use as a source of the missing utility gradient is an untested hypothesis and stays behind the go/no-go gate. Two efficiency items previously deferred for reasons that turned out to be wrong (AMP, conditional SDPA) are re-opened, with smaller expected payoffs than their headline numbers suggest.
A second pass over the literature identified the failure mode by name: optimizing a learned proxy against an expensive oracle produces a gold-score curve that rises, peaks, and then declines, which has been measured as reward-model overoptimization in RLHF and explains why the learned loss helped weak synthesizers and not strong ones. Best-of-n selection is consequently promoted from fallback to primary recommendation, since it has both an analytic optimization budget and a measured overoptimization curve, whereas gradient guidance has neither. Ensembling the estimator under a conservative scoring rule is the published mitigation. Dataset distillation is the adjacent field this work should be positioned against, and its distribution-matching branch is the one variant that does not require a differentiable downstream model.
A third pass, prioritising recent work, records that tabular foundation models now handle far larger tables than the figures quoted in the earlier passes, which makes regenerating the utility labels cheaper than assumed, though the leading model carries a non-commercial licence and its headline benchmark is not disinterested. It also identifies a published template for learning a usable gradient from a non-differentiable objective — a surrogate that is local, trained online alongside the optimisation, and continually resampled — none of which the estimator here currently does. Finally, the vendored synthesizers predate the current state of the art, so the claim that this method improves strong generators is being tested against a weaker definition of strong than now exists.
A fourth pass across meta-learning, transformers, and PyTorch practice found that the highest-value idea is also the oldest: landmarking, which characterises a table by the scores of a suite of deliberately weak, fast learners, costs seconds where the ground-truth utility label costs a full gradient-boosting fit. Those features both strengthen the cheap baseline that this work should be measured against and, in their linear form, supply a downstream model that is differentiable through the synthetic table. A 2026 study of metafeatures over fifty-one datasets found only weak signal surviving statistical control, which caps how far learned dataset embeddings can be pushed at this project's four datasets. The estimator is recognisably a neural process, a framing that supplies the uncertainty estimate the conservative-scoring mitigation requires. And current profiling guidance reorders the efficiency work: the data pipeline should be measured before any attention-kernel change, since an input-bound loop makes kernel optimisation worthless.
A fifth pass adds a constraint and a calibration. The constraint is environmental: this project targets Python 3.9, and PyTorch raised its minimum to 3.10 in late 2025, so the recent compiler work that would most help a small model with many small kernels is unreachable without an interpreter upgrade the pinned dependency forks may not survive. The calibration is more important. Recent theory decomposes the risk of an in-context predictor into a reducible gap, which depends on training data and is provably sensitive to distribution shift between training and deployment inputs, and an irreducible term set by the difficulty of the underlying task. The first formalises a shift this project already knew it had; the second means part of the estimator's error cannot be removed by any architecture. Measuring the ground-truth evaluator's own seed-to-seed variance establishes how much of the reported error is that floor; done 2026-08-08, see CLAUDE.md.
Taken together, the five passes point one way. Predicting a scalar quality score for a dataset is a solved problem type with several mature literatures behind it; extracting a usable descent direction from such a predictor is not, and every published method that manages it either requires a differentiable downstream model or surrounds the surrogate with machinery this project lacks. Selection by best-of-n requires no gradient, has a measured overoptimisation curve and an analytic budget, and should be the primary mode rather than the fallback. Four cheap measurements decide whether the rest matters, and all four are now done (2026-08-08, see CLAUDE.md): the ground-truth evaluator's seed-to-seed noise floor is a small fraction (1-16%) of cached label variance, but the cached labels turned out to use CatBoost's untuned epochs=1 default rather than a real fit; profiling the real training loop found it is not data-loading-bound (dataloader wait ≈0.02% of wall time), contrary to the fourth pass's prediction below, so kernel-level optimisation is not wasted effort; a random-forest-on-landmarkers baseline scores 0.78-0.97 held-out Spearman with no neural network, a bar the never-yet-run set-transformer estimator has to clear; and Gate A itself has now been run, for the first time, at toy scale — see below. Also done 2026-08-08 (see CLAUDE.md under "Quick wins implementation"): the pairwise ranking loss is rewritten to the LearningLoss++ form and on by default, the tf_num_inds=0 full-attention search hole is closed, SDPA and TF32 are on by default, and bf16 autocast plus per-epoch subsampling are opt-in — bf16 measured ~30x slower on this CPU-only dev box (no hardware bf16 support), so its CUDA payoff is still unverified here.
Best-of-n selection and ensemble conservative scoring, two of the "Big wins" the passes above point to, are implemented as reusable code (2026-08-08, ml_utility_loss/loss_learning/estimator/selection.py) — selection-time mechanisms that need no training run of their own, verified against mock and real (untrained) estimators, not yet wired into any synthesizer's sampling loop. The landmarker cheap-labeling idea is likewise promoted from a one-off script to ml_utility_loss/loss_learning/ml_utility/landmarkers.py:LandmarkerProxy. A ZeroGrads-style local surrogate (ml_utility_loss/loss_learning/estimator/local_surrogate.py:LocalOnlineSurrogate) is built and toy-tested, and Gate A (ml_utility_loss/loss_learning/evaluation/gate_a_tvae.py:run_gate_a) has now been run for the first time, end to end, at toy scale (scripts/gate_a_toy_run.py: tiny TVAE, tiny estimator, real CatBoost oracle) — a plumbing result (update_norm≈1e-6, win_rate=0, too few real-oracle labels for a meaningful gradient), not a real-scale verdict. Only a refreshed synthesizer baseline (TabSyn/GReaT) remains genuinely blocked: it needs vendoring new generator source code this environment has no path to, not a smaller run of code already here. Real-scale Gate A (a BEST-hyperparameter TVAE and estimator trained for real epochs) also remains GPU-blocked. See CLAUDE.md under "Big wins implementation" for the full writeup.
See CLAUDE.md for architecture and the detailed audit constraints, and AGENTS.md for development guidance.