Fraud detection and application PD are both rare-event classification problems, but they do not share one population, label process, cost model, or operating clock. Fraud is used in this repository as a contrast case: it shows why a generic instruction to rebalance the minority class is not enough.
An application-PD model often scores an applicant or application once and asks whether a defined credit event occurs within a performance horizon. A fraud system may score a transaction, session, device, claim, account action, or network edge in near real time.
That changes the experimental unit. Many fraud rows can belong to one person, account, merchant, or attack campaign. Random row splits can leak identity, campaign structure, or repeated patterns across train and test. Splits should respect time and the relevant entity or campaign boundary.
In a high-volume fraud stream, even a small false-positive rate can create an unmanageable number of alerts or blocked legitimate events. Aggregate ROC-AUC is therefore insufficient. Operating evidence should include:
- recall at a strict empirical FPR cap;
- precision and event prevalence for the evaluated stream;
- alerts per time unit and review capacity;
- review yield and time to action;
- value- or exposure-weighted capture where relevant;
- customer-friction measures for false alarms.
An interpolated ROC point is not itself an executable policy. The threshold, tie handling, achieved FPR, and predicted-positive volume must be reported.
The loss from a missed event may depend on amount, recoverability, downstream propagation, and intervention delay. A false alert may create review cost, customer friction, abandonment, or reputational harm. These values vary by case and action.
A single cost-sensitive threshold can be a transparent teaching example, but it should not be presented as a production fraud rule unless fixed, instance-independent costs and calibrated probabilities are genuinely appropriate. Capacity-constrained ranking or instance-level utility may be the better formulation.
Fraud labels can arrive through chargebacks, investigations, customer reports, manual review, or later network analysis. The observation process is selective: events that were blocked or reviewed are more likely to receive scrutiny than events allowed without investigation. Some confirmed negatives are merely uninvestigated events.
The dataset should distinguish, where possible:
- confirmed positive;
- confirmed negative;
- unresolved or not-yet-mature;
- policy-blocked without counterfactual outcome;
- reviewed versus unreviewed.
Collapsing all unresolved events into the negative class can overwhelm any gain from rebalancing with label noise and feedback bias.
Oversampling makes historical minority patterns more influential. In a static benchmark that may help a learner. In an adversarial stream it may also amplify campaigns that attackers have abandoned while missing a new mechanism.
Fraud experiments should therefore pair any sampling or weighting variant with:
- short, forward-only time splits;
- performance by attack or entity cluster where labels support it;
- feature and score drift monitoring;
- recall decay by event age;
- sensitivity to label-delay assumptions;
- challenger evaluation on genuinely later data.
The question is not only whether recall rises, but whether the added recall persists after the fraud mechanism changes.
An event probability can be useful for expected loss or queue prioritization, but its base rate may change rapidly by channel, merchant, geography, attack campaign, or control policy. Prior correction under a stable prior-shift model is a narrow tool; repeated calibration monitoring is still required.
If the score is used only to fill a fixed review queue, ranking and capacity may be the primary contract. Calling that score a probability adds a claim that must be validated separately.
Fraud controls change attacker behavior and the labels the system receives. Blocking an event prevents its natural outcome; reviewing an event increases its chance of being labeled; an attacker adapts to the deployed controls. These feedback paths can invalidate an offline comparison even when its random-split metrics look strong.
Monitoring should therefore separate:
- incoming-stream prevalence and composition;
- score distribution and alert volume;
- review selection and adjudication rate;
- mature label yield by policy path;
- model performance on comparable, label-mature cohorts;
- emerging clusters not represented in training.
- Define the event unit, label sources, maturity delay, and entity linkage.
- Use forward time splits with entity/campaign leakage controls.
- Keep validation and test streams at their observed prevalence.
- Compare an unrebalanced baseline with a small number of justified training- signal variants.
- Select thresholds under an explicit FPR or capacity constraint on validation.
- Report achieved FPR, recall, precision, alert volume, and review yield on a later frozen cohort.
- Stress test drift, label delay, unresolved cases, and amount-weighted loss.
- Document which labels are created by the existing policy.
The reusable code supports binary metrics and empirical operating points, but it does not encode a universal fraud cost model. See metrics and operating points. The synthetic fixed-FPR example demonstrates the narrow empirical-threshold contract without claiming a production policy.