This protocol is designed for a public, synthetic retail-credit example and for internal adaptation to properly governed data. It compares training signals without confusing them with target-population, calibration, or policy changes.
Write one primary hypothesis before tuning. Examples:
- class weighting improves out-of-time recall under an FPR cap without materially degrading calibration after recalibration;
- a training-only sampler improves PR-AUC relative to an equal-budget baseline;
- focal loss changes hard-case ranking but requires a separate probability calibration step.
Also state the estimand, event horizon, target population, and primary metric. "Find the best imbalance method" is too broad to falsify and encourages metric shopping.
Document the full funnel from applications to label-mature accounts. Freeze:
- feature availability at the application date;
- observation and performance windows;
- event, cure, and maturity definitions;
- historical policy, approval, booking, and data-quality filters;
- duplicate/entity handling;
- train, validation, test, and out-of-time cutoff dates.
Report event counts and prevalence for every unresampled cohort. Do not describe approved/booked mature outcomes as the full application population.
A defensible default is:
- train: fit preprocessing, samplers, estimators, and candidate models;
- validation: tune hyperparameters, fit a calibrator where needed, and select an operating point;
- test: estimate performance of the frozen model/calibrator/rule;
- out-of-time: assess temporal transport and policy stability.
Where cross-validation is used inside the training era, all learned feature transformations and sampling steps are fitted inside each fold. Entity-level or household leakage must also be controlled when multiple rows can refer to a related borrower or account.
At minimum compare:
- unrebalanced baseline;
- class-weighted variant;
- one justified training-only sampler;
- one loss-level variant if supported by the estimator;
- the same ranking model with post-fit operating-point selection.
Use the same feature set, learner family, search budget, early-stopping rule, and random-seed policy unless a difference is part of the hypothesis. Record sampling ratios, weight formulas, and effective event counts.
Do not compare an extensively tuned rebalanced model with an untuned baseline.
For every fold or training split:
raw training rows
-> training-fitted preprocessing
-> training-only sampler (if any)
-> estimator
Validation, test, and out-of-time rows pass through the fitted preprocessing and
estimator but never through fit_resample. This is both a leakage control and a
population-definition control.
Report ROC-AUC, KS, PR-AUC with prevalence, and the primary operating-region metric. If an FPR cap is material, report the achieved empirical FPR, recall, threshold, predicted-positive rate, and tie rule.
For outputs claimed as PD, report Brier score and reliability by score band, time, and important segments. Explain whether known-prior correction was used, which assumptions support it, and where calibration was fitted.
Declare the cost model or operational constraint. Select the cutoff on validation data only. Report confusion counts, approval/review rates, expected cost, and sensitivity to plausible cost or capacity ranges.
Evaluate approval and booking rates, expected/observed bad rate, exposure or risk mix, review workload, and relevant guardrails. A metric gain that depends on an unacceptable population shift is not a clean model gain.
Point estimates are fragile when events are rare. Use resampling or another appropriate method to report uncertainty for the primary comparison, while respecting time and entity structure. Report absolute differences as well as relative percentages.
Repeat stochastic pipelines across declared seeds. Examine:
- time-cohort stability;
- channel/product/segment stability where support is adequate;
- sensitivity to sampling ratio or class weight;
- calibration drift after prevalence changes;
- operating-point sensitivity near score ties;
- performance for hypothesized bad-outcome strata, labeled as exploratory when not pre-specified.
Avoid declaring subgroup conclusions from tiny event counts.
The test and out-of-time cohorts are not tuning dashboards. Once inspected, changes motivated by their results require a new final holdout or must be reported as exploratory.
This applies to feature selection, rebalancing ratios, calibration methods, thresholds, subgroup definitions, and narrative choices about which metric is primary.
Each row should identify one frozen pipeline. Suggested columns are:
| Field group | Required fields |
|---|---|
| Cohort | dates, maturity rule, row/event counts, prevalence |
| Training signal | sampler and ratio, class/instance weights, loss parameters |
| Ranking | ROC-AUC, KS, PR-AUC, recall and achieved FPR at the chosen cap |
| Probability | correction/calibrator, Brier score, reliability summary |
| Decision | rule source, threshold, action rate, confusion counts, expected cost |
| Portfolio | approval/review rate, bad-rate or loss estimate, guardrail results |
| Robustness | seed dispersion, out-of-time change, important caveats |
A repository example should make the data-generating assumptions visible:
- an application-era feature matrix with no real customer fields;
- a historical approval mechanism that creates selected booked data;
- delayed label maturity;
- a rare bad outcome with configurable prevalence;
- a time shift for out-of-time evaluation;
- no claim that the synthetic values reproduce a bank portfolio.
The example's purpose is to demonstrate contracts and failure modes, not to advertise production accuracy.
The repository implements this design in the synthetic credit-risk workflow and keeps the observation boundary visible in the reject-inference example.
Fraud studies should add review capacity, selective label feedback, rapid time splits, and drift/adversarial checks. Their event definition and negative class may mature differently from application PD. See The fraud boundary and the fixed-FPR synthetic example.
From a fresh environment, the public package should install, run tests, build, install its wheel in isolation, and run every public example. The project-level commands are listed in the project overview. Exact numerical outputs should be documented only after they have been verified by the current test and example suite.