Skip to content

Commit 7c9a653

Browse files
feat: regression analysis path + TrustReport integration (#82) (#142)
* feat: regression analysis path + TrustReport integration (#82) Phase 1 of regression support, following the maintainer's preferred ordering (pipeline/report first; visualization and a regression trust score to follow). - Task routing: analyze(..., task="auto"|"classification"|"regression"). Auto-detection errs toward classification (only clearly-continuous float targets route to regression, so discrete label sets are never mis-routed). Regression skips the classification backend (no class probabilities) and resolves point predictions from y_pred or model.predict(X). - Regression pipeline (_run_regression_pipeline) mirrors the classification pipeline (metrics -> results -> report), routing through the merged regression metrics: error_distribution, prediction_interval_coverage (PICP), error_variance_correlation. Uncertainty metrics degrade gracefully when prediction_intervals / predicted_variance are not supplied. - TrustReport gains task_type. For regression it surfaces the metrics in results["regression"], renders a dedicated show() report, and serializes via to_dict()/save() -- without the classification Trust Score (the regression score is a separate phase). Classification-only features (summary_plot, plot, plot_bias, show_failures, deployment_explanation) raise a clear NotImplementedError on regression reports. Regression logic is isolated; the classification path is unchanged. Tests: tests/test_regression_pipeline.py (10) -- routing, auto-detection, show(), to_dict(), save() round-trip, model.predict path, uncertainty metrics, and the classification-only guards. Full suite green; ruff + mypy (--follow-imports=skip) clean. Refs #82. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address automated review on regression path (routing, shape, style) - api.py: integer-valued float targets route to classification at ANY cardinality (a 25-class label set encoded as float was mis-routed to regression); only clearly-continuous floats go to regression. - pipeline.py: accept (n, 1) singleton-column predictions/intervals/variance by flattening; reject true multi-output with a clear error (was a latent shape-mismatch crash). Use the verbose-aware logger, not raw print(). - report.py: move io/redirect_stdout imports to module top (out of methods); enrich regression to_dict() with n_samples/model/timestamp for parity with the classification serialization. - tests: +3 (multi-class float routing, singleton-column flatten + multi-output rejection, regression to_dict metadata). Suite 355 -> 358. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style: apply ruff format to regression-path files CI `ruff format` pre-commit hook reformatted the 4 files touched by the regression path; bringing them to ruff's canonical style so the Linting check passes. Whitespace/line-wrap only — no logic change. Suite 358 green, ruff check + mypy --follow-imports=skip clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Whatsonyourmind <248369826+Whatsonyourmind@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 104f3b3 commit 7c9a653

4 files changed

Lines changed: 578 additions & 10 deletions

File tree

‎tests/test_regression_pipeline.py‎

Lines changed: 212 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,212 @@
1+
"""Tests for the regression analysis path: task routing + TrustReport integration.
2+
3+
Covers analyze() task detection/routing, regression metric surfacing in the
4+
report, show() rendering, to_dict() serialization, save() round-trips, and that
5+
classification-only features are cleanly guarded on a regression report.
6+
"""
7+
8+
from __future__ import annotations
9+
10+
import numpy as np
11+
import pytest
12+
13+
from trustlens import analyze
14+
from trustlens.report import TrustReport
15+
16+
17+
@pytest.fixture
18+
def regression_data():
19+
rng = np.random.default_rng(42)
20+
n = 200
21+
X = rng.normal(size=(n, 3))
22+
y_true = X @ np.array([1.5, -2.0, 0.7]) + rng.normal(scale=1.0, size=n)
23+
y_pred = y_true + rng.normal(scale=1.0, size=n)
24+
return X, y_true, y_pred
25+
26+
27+
def test_explicit_regression_routes_and_skips_trust_score(regression_data):
28+
X, y_true, y_pred = regression_data
29+
rep = analyze(model=None, X=X, y_true=y_true, y_pred=y_pred, task="regression", verbose=False)
30+
assert isinstance(rep, TrustReport)
31+
assert rep.task_type == "regression"
32+
assert rep.trust_score is None # classification score intentionally absent
33+
ed = rep.results["regression"]["error_distribution"]
34+
for key in (
35+
"mean_absolute_error",
36+
"rmse",
37+
"median_absolute_error",
38+
"p90_absolute_error",
39+
"max_error",
40+
):
41+
assert key in ed
42+
assert rep.metadata["task_type"] == "regression"
43+
assert "n_classes" not in rep.metadata
44+
assert rep.metadata["n_unique_targets"] == len(np.unique(y_true))
45+
46+
47+
def test_auto_detection(regression_data):
48+
X, y_true, y_pred = regression_data
49+
# Continuous target -> regression.
50+
rep = analyze(model=None, X=X, y_true=y_true, y_pred=y_pred, task="auto", verbose=False)
51+
assert rep.task_type == "regression"
52+
# Integer label target -> classification (not routed to regression).
53+
y_cls = (y_true > np.median(y_true)).astype(int)
54+
yprob = np.column_stack([1 - y_cls, y_cls]).astype(float)
55+
rep_cls = analyze(
56+
model=None, X=X, y_true=y_cls, y_pred=y_cls, y_prob=yprob, task="auto", verbose=False
57+
)
58+
assert rep_cls.task_type == "classification"
59+
60+
61+
def test_show_renders_regression(regression_data, capsys):
62+
X, y_true, y_pred = regression_data
63+
rep = analyze(model=None, X=X, y_true=y_true, y_pred=y_pred, task="regression", verbose=False)
64+
rep.show()
65+
out = capsys.readouterr().out
66+
assert "Regression Reliability Report" in out
67+
assert "MAE" in out and "RMSE" in out
68+
assert "Task : regression" in out
69+
70+
71+
def test_to_dict_serializes_regression(regression_data):
72+
X, y_true, y_pred = regression_data
73+
rep = analyze(model=None, X=X, y_true=y_true, y_pred=y_pred, task="regression", verbose=False)
74+
d = rep.to_dict()
75+
assert d["task_type"] == "regression"
76+
assert "regression.error_distribution.rmse" in d
77+
assert isinstance(d["regression.error_distribution.rmse"], float)
78+
# No classification trust score leaks into a regression dict.
79+
assert "trust_score" not in d
80+
assert "trust_grade" not in d
81+
82+
83+
def test_uncertainty_metrics_populated_when_supplied(regression_data):
84+
X, y_true, y_pred = regression_data
85+
lo, hi = y_pred - 2.0, y_pred + 2.0
86+
variance = np.abs(y_true - y_pred) # perfectly tracks error
87+
rep = analyze(
88+
model=None,
89+
X=X,
90+
y_true=y_true,
91+
y_pred=y_pred,
92+
task="regression",
93+
prediction_intervals=(lo, hi),
94+
predicted_variance=variance,
95+
confidence_level=0.95,
96+
verbose=False,
97+
)
98+
reg = rep.results["regression"]
99+
picp = reg["interval_coverage"]
100+
assert "picp" in picp and 0.0 <= picp["picp"] <= 1.0
101+
assert picp["target_coverage"] == 0.95
102+
corr = reg["error_variance_correlation"]
103+
assert corr["verdict"] == "informative" # variance == error => strong corr
104+
105+
106+
def test_uncertainty_metrics_skip_gracefully(regression_data):
107+
X, y_true, y_pred = regression_data
108+
rep = analyze(model=None, X=X, y_true=y_true, y_pred=y_pred, task="regression", verbose=False)
109+
reg = rep.results["regression"]
110+
assert reg["interval_coverage"]["status"] == "skipped"
111+
assert reg["error_variance_correlation"]["status"] == "skipped"
112+
113+
114+
def test_model_predict_path():
115+
from sklearn.linear_model import LinearRegression
116+
117+
rng = np.random.default_rng(0)
118+
n = 120
119+
X = rng.normal(size=(n, 4))
120+
y = X @ np.array([1.0, 0.5, -1.0, 2.0]) + rng.normal(scale=0.5, size=n)
121+
model = LinearRegression().fit(X, y)
122+
# No y_pred given -> resolved via model.predict.
123+
rep = analyze(model=model, X=X, y_true=y, task="regression", verbose=False)
124+
assert rep.task_type == "regression"
125+
assert rep.results["regression"]["error_distribution"]["n_samples"] == n
126+
127+
128+
def test_classification_features_guarded_on_regression(regression_data):
129+
X, y_true, y_pred = regression_data
130+
rep = analyze(model=None, X=X, y_true=y_true, y_pred=y_pred, task="regression", verbose=False)
131+
for call in (
132+
lambda: rep.summary_plot(show=False),
133+
lambda: rep.plot(),
134+
lambda: rep.show_failures(),
135+
lambda: rep.deployment_explanation,
136+
):
137+
with pytest.raises(NotImplementedError):
138+
call()
139+
140+
141+
def test_save_roundtrip_regression(regression_data, tmp_path):
142+
import json
143+
144+
X, y_true, y_pred = regression_data
145+
rep = analyze(model=None, X=X, y_true=y_true, y_pred=y_pred, task="regression", verbose=False)
146+
# JSON
147+
jp = rep.save(str(tmp_path / "reg.json"))
148+
data = json.loads(jp.read_text(encoding="utf-8"))
149+
assert data["task_type"] == "regression"
150+
assert "regression" in data["results"]
151+
# TXT
152+
tp = rep.save(str(tmp_path / "reg.txt"))
153+
assert "Regression Reliability Report" in tp.read_text(encoding="utf-8")
154+
155+
156+
def test_invalid_task_raises(regression_data):
157+
X, y_true, y_pred = regression_data
158+
with pytest.raises(ValueError):
159+
analyze(model=None, X=X, y_true=y_true, y_pred=y_pred, task="nonsense", verbose=False)
160+
161+
162+
def test_multiclass_float_labels_route_to_classification():
163+
"""A 25-class label set encoded as float must NOT be mistaken for
164+
regression — integer-valued floats are class labels at any cardinality."""
165+
rng = np.random.default_rng(7)
166+
n = 300
167+
X = rng.normal(size=(n, 3))
168+
y_cls = rng.integers(0, 25, size=n).astype(float) # 25 integer classes as float
169+
assert len(np.unique(y_cls)) > 20
170+
yprob = np.full((n, 25), 1 / 25)
171+
rep = analyze(
172+
model=None, X=X, y_true=y_cls, y_pred=y_cls, y_prob=yprob, task="auto", verbose=False
173+
)
174+
assert rep.task_type == "classification"
175+
176+
177+
def test_singleton_column_predictions_flattened():
178+
"""`(n, 1)` predictions (valid single-output) flatten rather than crash;
179+
a true multi-output shape is rejected with a clear error."""
180+
rng = np.random.default_rng(11)
181+
n = 150
182+
X = rng.normal(size=(n, 2))
183+
y_true = (X @ np.array([1.0, -0.5])) + rng.normal(scale=0.3, size=n)
184+
y_pred_2d = (y_true + rng.normal(scale=0.3, size=n)).reshape(-1, 1) # (n, 1)
185+
rep = analyze(
186+
model=None,
187+
X=X,
188+
y_true=y_true.reshape(-1, 1),
189+
y_pred=y_pred_2d,
190+
task="regression",
191+
verbose=False,
192+
)
193+
assert rep.results["regression"]["error_distribution"]["n_samples"] == n
194+
# True multi-output is rejected, not silently mis-shaped.
195+
with pytest.raises(ValueError):
196+
analyze(
197+
model=None,
198+
X=X,
199+
y_true=y_true,
200+
y_pred=np.column_stack([y_pred_2d[:, 0], y_pred_2d[:, 0]]),
201+
task="regression",
202+
verbose=False,
203+
)
204+
205+
206+
def test_to_dict_includes_regression_metadata(regression_data):
207+
X, y_true, y_pred = regression_data
208+
rep = analyze(model=None, X=X, y_true=y_true, y_pred=y_pred, task="regression", verbose=False)
209+
d = rep.to_dict()
210+
assert d["n_samples"] == len(y_true)
211+
assert d["model"] == "Manual"
212+
assert "timestamp" in d and "trustlens_version" in d

‎trustlens/api.py‎

Lines changed: 77 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,41 @@
2424
import numpy as np
2525

2626
from trustlens.backends.registry import get_resolver
27-
from trustlens.core.pipeline import _run_analysis_pipeline
27+
from trustlens.core.pipeline import _run_analysis_pipeline, _run_regression_pipeline
2828
from trustlens.report import TrustReport
2929

3030
logger = logging.getLogger(__name__)
3131

3232

33+
def _detect_task(y_true: np.ndarray, task: str) -> str:
34+
"""Resolve the analysis task type.
35+
36+
``task`` may be ``"classification"`` / ``"regression"`` (explicit, honored
37+
as-is) or ``"auto"``. Auto-detection errs toward ``"classification"`` and
38+
only returns ``"regression"`` when the target is clearly continuous — a
39+
float array that is not integer-valued, or has many distinct values — so a
40+
discrete label set is never mis-routed.
41+
"""
42+
if task in ("classification", "regression"):
43+
return task
44+
if task != "auto":
45+
raise ValueError(f"Invalid task {task!r}. Use 'auto', 'classification', or 'regression'.")
46+
47+
y = np.asarray(y_true)
48+
if y.dtype.kind == "f":
49+
n_unique = len(np.unique(y))
50+
is_integer_valued = bool(np.all(np.isfinite(y))) and bool(np.allclose(y, np.round(y)))
51+
# Integer-valued floats are class labels at ANY cardinality (a 25-class
52+
# target encoded as float must not be mistaken for regression), and a
53+
# small distinct-value set is also label-like. Only clearly-continuous
54+
# floats route to regression.
55+
if is_integer_valued or n_unique <= 20:
56+
return "classification"
57+
return "regression"
58+
# Non-float dtypes (ints, strings, bools) default to classification.
59+
return "classification"
60+
61+
3362
def quick_analyze(
3463
model=None, X=None, y=None, dataset="iris", framework: Optional[str] = None
3564
) -> TrustReport:
@@ -107,6 +136,10 @@ def analyze(
107136
modules: Optional[list[str]] = None,
108137
plugins: Optional[list[str]] = None,
109138
class_labels: Optional[np.ndarray] = None,
139+
task: str = "auto",
140+
prediction_intervals: Optional[tuple[np.ndarray, np.ndarray]] = None,
141+
predicted_variance: Optional[np.ndarray] = None,
142+
confidence_level: float = 0.95,
110143
verbose: bool = True,
111144
) -> TrustReport:
112145
"""
@@ -143,6 +176,20 @@ def analyze(
143176
Semantic class labels in the order corresponding to probability columns.
144177
Useful for raw backends such as ``xgboost.Booster`` that return ordinal
145178
probability columns without a ``classes_`` attribute.
179+
task : str, default='auto'
180+
Analysis task: ``'auto'`` (detect from ``y_true``), ``'classification'``,
181+
or ``'regression'``. Regression routes through the regression reliability
182+
metrics (error distribution, interval coverage, error-variance
183+
correlation) instead of the classification modules.
184+
prediction_intervals : tuple(np.ndarray, np.ndarray), optional
185+
``(lower, upper)`` per-sample prediction-interval bounds (regression only).
186+
Enables Prediction Interval Coverage (PICP); omitted ⇒ that metric is
187+
skipped.
188+
predicted_variance : np.ndarray, optional
189+
Per-sample predicted variance / uncertainty score (regression only).
190+
Enables the error-variance correlation metric; omitted ⇒ skipped.
191+
confidence_level : float, default=0.95
192+
Nominal coverage the supplied ``prediction_intervals`` claim (regression).
146193
verbose : bool
147194
Print progress updates. Default True.
148195
@@ -184,7 +231,35 @@ def analyze(
184231
>>> report.show()
185232
"""
186233
if len(y_true) < 30:
187-
logger.warning("Small dataset (n < 30) detected. Calibration metrics may be unreliable.")
234+
logger.warning("Small dataset (n < 30) detected. Metrics may be unreliable.")
235+
236+
# ------------------------------------------------------------------
237+
# 0. Route by task. Regression skips the classification backend (which
238+
# resolves class probabilities) and the classification modules.
239+
# ------------------------------------------------------------------
240+
task_type = _detect_task(y_true, task)
241+
if task_type == "regression":
242+
if y_pred is None:
243+
if model is None or not hasattr(model, "predict"):
244+
raise ValueError(
245+
"Regression analysis needs point predictions: pass y_pred=..., "
246+
"or a model that exposes .predict(X)."
247+
)
248+
y_pred_resolved = np.asarray(model.predict(X))
249+
else:
250+
y_pred_resolved = np.asarray(y_pred)
251+
return _run_regression_pipeline(
252+
model=model,
253+
X=X,
254+
y_true=np.asarray(y_true),
255+
y_pred=y_pred_resolved,
256+
prediction_intervals=prediction_intervals,
257+
predicted_variance=predicted_variance,
258+
confidence_level=confidence_level,
259+
framework=framework or ("manual" if y_pred is not None else None),
260+
backend_metadata={"task_type": "regression"},
261+
verbose=verbose,
262+
)
188263

189264
# ------------------------------------------------------------------
190265
# 1. Resolve predictions via Backend Registry

0 commit comments

Comments
 (0)