ranking-calibration-decision.md

Ranking, calibration, decision, and portfolio

Languages: English · 简体中文

One fitted model can be discussed as if it produced one answer. In credit risk it usually participates in at least four different tasks:

1. rank accounts by relative risk;
2. estimate an event probability for a stated population and horizon;
3. convert a score or probability into an action;
4. shape the risk and economics of the resulting portfolio.

An improvement at one layer does not prove an improvement at the next.

1. Ranking: who is placed ahead of whom?

A ranking score needs a stable ordering, not a probability scale. Any strictly increasing transformation preserves the ordering and therefore preserves rank-based metrics, while changing the numerical score and every fixed numeric cutoff.

Useful ranking evidence includes:

• ROC-AUC or Gini for pairwise ordering;
• KS for maximum empirical separation;
• PR-AUC when the event is rare, always reported with event prevalence;
• recall under a declared false-positive-rate cap;
• capture within a top score band or review capacity;
• stability by time, channel, product, and relevant applicant segment.

No single value answers every operating question. ROC-AUC can remain acceptable while the extreme low-FPR region deteriorates. PR-AUC changes with prevalence, so values from differently selected cohorts are not directly interchangeable.

Empirical operating points

When a policy says "FPR must not exceed 1%," the deployable object is an empirical score threshold, not an interpolated point drawn between two ROC coordinates. Ties must be handled as a group because identical scores cannot be split by a deterministic threshold alone.

An operating-point procedure should declare:

• whether the constraint is FPR <= cap or nearest-to-cap;
• how score ties are handled;
• what happens when no positive prediction is feasible under the cap;
• which cohort was used to select the threshold;
• whether a separate test or out-of-time cohort confirms it.

The package keeps historical weighted coverage and deployable FPR-cap selection as separate contracts. Historical reproduction should not silently inherit a new interpolation rule.

2. Calibration: what population does the number describe?

A calibrated application PD has an absolute interpretation, for example: among comparable applicants assigned 0.08, roughly 8% experience the defined event within the stated horizon, subject to sampling uncertainty and stability.

That statement depends on:

• the target population and selection policy;
• the event and performance-window definition;
• the time period and covariate distribution;
• the training sampling or weighting scheme;
• the calibration sample and method.

A rank-preserving recalibration can improve probability meaning without materially changing ranking. Conversely, a rebalanced learner can improve ranking while making its raw score unsuitable as PD.

Known-prior correction is conditional, not magic

Under case-control sampling with known sampling rates, unchanged class- conditional feature distributions, and prior-probability shift, a sampled posterior can be adjusted through prior odds. If q(x) is the sampled posterior, pi_s the sampled event prevalence, and pi_t the target prevalence:

target_odds(x)
  = q(x) / (1 - q(x))
    * [pi_t / (1 - pi_t)] / [pi_s / (1 - pi_s)]

This relation does not establish that a fitted score is a calibrated sampled posterior, nor does it repair covariate shift, policy selection, synthetic sampling artifacts, model misspecification, or reject-label non-observability. It is a correction under stated assumptions, followed by independent calibration checks.

Calibration should be assessed on an unresampled, target-like validation cohort and confirmed on a held-out or out-of-time cohort. Report reliability by score band, Brier score, and segment/time stability; avoid relying on one aggregate plot.

3. Decision: which action minimizes a declared loss?

A decision rule maps model output plus constraints to an action such as approve, decline, refer, or request additional evidence. It is not part of ranking by default.

If p is a calibrated probability for the decision population, and the four classification costs are fixed and instance-independent, the positive action is preferred at or above:

threshold
  = (C_FP - C_TN)
    / [(C_FN - C_TP) + (C_FP - C_TN)]

provided each misclassification increment is non-negative and the denominator is positive. With zero correct-classification costs, this reduces to C_FP / (C_FP + C_FN).

The formula is useful because its assumptions are explicit. It should not be applied directly to an arbitrary ranking score. It also becomes inadequate when costs vary by exposure, limit, margin, term, collection path, customer value, or regulatory treatment. Those settings require instance-level economics or a portfolio optimization, not one universal cutoff.

Other valid decision constraints include:

• minimum approval rate;
• maximum expected loss or bad rate;
• review or verification capacity;
• an FPR cap for a defined negative population;
• channel/product guardrails;
• risk-return or capital constraints.

Thresholds are selected on validation data after the model and any calibrator have been fitted. Test data should estimate the performance of the already selected rule, not participate in selecting it.

4. Portfolio: what population does the policy create?

Account-level actions accumulate into a portfolio. Relevant outcomes include:

• approval and booking rates;
• expected and realized bad rates;
• expected loss and risk-adjusted return;
• exposure, term, and segment mix;
• operational workload and review yield;
• stability across time and acquisition channels.

A policy can improve a model metric while producing an undesirable portfolio mix. It can also change which labels become observable in the next training cycle. This is a feedback loop: decisions shape future booked populations, outcomes, and data support.

Champion/challenger analysis should therefore preserve the model layer and the policy layer separately. Compare ranking on common cohorts, calibration for the stated population, and portfolio outcomes under a documented action rule.

5. A disciplined sequence

For each candidate model:

1. define population, horizon, label, and maturity;
2. evaluate ranking on unresampled validation and out-of-time cohorts;
3. correct known prior changes where assumptions support it;
4. fit and validate calibration without using test outcomes;
5. select an operating point from explicit costs or constraints;
6. evaluate the frozen rule on held-out data;
7. simulate portfolio effects and policy feedback;
8. monitor ranking, calibration, action rates, and population drift separately.

The corresponding code contracts are described in metrics, operating points, probability, and decision. The synthetic credit-risk workflow composes all four layers, while the fraud example isolates an empirical FPR-cap operating point.