Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
6157ce6
Lazy experiment lifecycle: configure → optional prior checks → fit()
drbenvincent Aug 22, 2026
232b8b3
Migrate test suite to the lazy lifecycle API
drbenvincent Aug 22, 2026
b1503be
Document the lazy lifecycle in ARCHITECTURE.md, skill, release notes
drbenvincent Aug 22, 2026
e98c1a1
Address round-1 review findings
drbenvincent Aug 22, 2026
d801d33
Guard BBETS build() against post-conditioning node poisoning and pin …
drbenvincent Aug 22, 2026
c7c2e07
Pin the BBETS re-arm path in the refit regression test
drbenvincent Aug 22, 2026
120981d
Cover the prior-phase contract; fix two latent bugs surfaced by coverage
drbenvincent Aug 24, 2026
e8caf65
Migrate docs notebook corpus to the lazy lifecycle API
drbenvincent Aug 24, 2026
859df08
Pin the new JAX-dependent notebook skip in the consistency test
drbenvincent Aug 24, 2026
021a0af
Fix doubled result-bundle prefixes in five notebooks
drbenvincent Aug 24, 2026
20351ab
Rename GroupNotSampleedException to GroupNotSampledException
drbenvincent Aug 24, 2026
1a5f0f5
Type the lifecycle precisely: generic result bundles and deduped assi…
drbenvincent Aug 24, 2026
5753122
Honor the requested draw group in PanelRegression plots
drbenvincent Aug 24, 2026
bb15741
Keep stub-plot experiments reportable
drbenvincent Aug 24, 2026
04a2d58
Document lifecycle forwarders under Other Parameters
drbenvincent Aug 24, 2026
410dbc4
Deduplicate result-bundle fallback in maketables adapters
drbenvincent Aug 24, 2026
2964544
Fix ARCHITECTURE drift and document the IV build() exception
drbenvincent Aug 24, 2026
af0830d
Fix second-pass review findings: IV fit kwargs, refit ppc, prior pred…
drbenvincent Aug 24, 2026
b83a015
Align remaining test fakes and IV forwarder fragments
drbenvincent Aug 24, 2026
4b064b8
Pin prior-phase defaults, guard rebuild-with-changed-inputs, silence …
drbenvincent Aug 24, 2026
7e1a63c
Address final review: dedupe PrePostNEGD counterfactual, persist IV p…
drbenvincent Aug 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,15 @@ The optional third backend, `PyMCForecastModel` (`causalpy/pymc_forecast_models.

## Experiment Lifecycle

Instantiation fits eagerly in `__init__`: `_build_design_matrices()` → `_prepare_data()` → `algorithm()`. There is no separate `.fit()` on the experiment. Each subclass's public `plot(*, ...)` delegates to `_render_plot()`, which calls the subclass's backend-agnostic `_plot()`. Uncertainty rendering keys on data properties of the canonical prediction container (`has_posterior_draws()`), not on backend identity. `effect_summary()` returns `EffectSummary(table, text)` using helpers from `causalpy.reporting`.
Construction is lazy: `__init__` runs validation and deterministic preprocessing only (`_build_design_matrices()` → `_prepare_data()`); nothing is sampled. The lifecycle is `configure → optional prior checks → fit()`:

- **`build()`** — public, idempotent, auto-called by both samplers. Merges data-driven priors (defaults → data-derived → user) and constructs the PyMC graph so the spec is inspectable (`pm.model_to_graphviz(exp.model)`, `exp.model.basic_RVs`) before any compute is spent.
- **`sample_prior_predictive(**kwargs)`** — optional prior phase; populates `exp.prior_result`. Raises `PriorPredictiveNotSupportedException` on backends without a prior phase (sklearn, pymc-forecast, IV, state space).
- **`fit(**kwargs)`** — posterior phase; runs NUTS + posterior predictive first (forward-sampling machinery mutates shared data nodes, so posterior must sample on freshly armed data), then fills the absent prior phase so `idata` matches the historical eager output. Returns `Self`, so call sites migrate with one appended token: `cp.InterruptedTimeSeries(...).fit()`.

Draw-derived state lives in per-experiment bundles (`causalpy/experiments/_results.py`) exposed through two raising properties: `exp.result` (posterior group) and `exp.prior_result` (prior group). Nothing derived from draws is assigned to the experiment itself; `is_fitted` / `has_prior_predictive` are predicates over those two slots. Read methods take keyword-only `group: Literal["prior", "posterior"]`; the guard and group→bundle resolution live once in `_render_plot()` / `_resolve_group()`. Missing groups raise `GroupNotSampledException` naming the call to make; both exceptions are re-exported from `causalpy`. Re-sampling overwrites its own group only (`fit()` again replaces posterior and warns; prior state survives). Assigning a new model — also the documented prior-revision path instead of a `set_priors()` — resets everything, because graph identity *is* the model instance.

Each subclass's public `plot(*, ...)` delegates to `_render_plot()`, which calls the subclass's backend-agnostic `_plot(group=..., ...)`. Uncertainty rendering keys on data properties of the canonical prediction container (`has_posterior_draws()`), not on backend identity. `plot(group="prior")` renders a reduced panel set (counterfactual vs observed only). `effect_summary(group=...)` returns `EffectSummary(table, text)` using helpers from `causalpy.reporting`; prior-group prose reads as a plausibility check, not a causal claim.

## Experiment Inventory

Expand Down Expand Up @@ -98,21 +106,20 @@ Instantiation fits eagerly in `__init__`: `_build_design_matrices()` → `_prepa

| Topic | Detail |
|-------|--------|
| **Lazy fitting** | `__init__` never samples. Explicit `fit()` / `sample_prior_predictive()` run the phases; see Experiment Lifecycle. |
| **Formulas** | Patsy `dmatrices()` for design matrices; `build_design_matrices()` for counterfactual prediction. Bare datetime predictors are encoded as continuous elapsed days from the fitted origin; use `C(date)` for date fixed effects. `PiecewiseITS` uses `step()`/`ramp()` stateful transforms. |
| **obs_ind** | All experiments set `data.index.name = "obs_ind"`. Canonical xarray/PyMC dimension name. |
| **treated_units always 2D** | Even single-unit experiments use `treated_units=["unit_0"]`. Never pass 1D y to PyMC. |
| **Impact uses mu, not y_hat** | The adapter's `predict()` extracts posterior `mu` (conditional expected outcome in observed units), not `y_hat` (with observation noise); impact is `y - predict(X)`. For GLMs, `mu` must be inverse-linked before impact; see `docs/source/knowledgebase/prediction-contract.md`. |
| **Intercept handling** | Patsy includes intercept by default. sklearn models must use `fit_intercept=False`. |
| **Eager fitting** | MCMC runs during `__init__`. No lazy `.fit()` on the experiment. |
| **HDI_PROB** | Project default is 0.94 (ArviZ default), not 0.95. |
| **create_causalpy_compatible_class** | Applied during `make_model_adapter()` for sklearn backends; clones the user instance before patching. |

## Adding New Code

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The HDI_PROB row got dropped from this table in the rewrite. Base had:

| **HDI_PROB** | Project default is 0.94 (ArviZ default), not 0.95. |

That's still true, still surprising (most people assume 0.95), and this was the only place it was written down. Looks like collateral from removing the "Eager fitting" row right next to it.

Fix: put the row back.


Copy the closest existing experiment or model and follow the `BaseExperiment` contract:

- Declare `supports_ols` / `supports_bayes` (and `supports_pymc_forecast` to opt into the optional pymc-forecast backend); implement a single backend-agnostic `_plot()` (and an explicit `get_plot_data(*, ...)` only where that view is supported) that consumes the canonical prediction container, keying uncertainty rendering on `has_posterior_draws()` rather than backend identity
- `algorithm()` with the fit/predict/impact flow; every concrete experiment declares its own explicit `effect_summary(...)` contract, using helpers in `causalpy.reporting` where that summary is implemented
- Implement `_fit_inputs()` returning the `(X, y, coords)` handed to the backend at build time, and `_finalize(group)` computing the experiment's result bundle from the group's draws; every concrete experiment declares its own explicit `effect_summary(*, group=..., ...)` contract, using helpers in `causalpy.reporting` where that summary is implemented
- Declare `supports_ols` / `supports_bayes` (and `supports_pymc_forecast` when the experiment accepts a `PyMCForecastModel`) plus `_default_model_class`; `make_model_adapter()` validates them at construction. Subscript the base with the experiment's bundle type (e.g. `BaseExperiment[CausalResult]`, or `BaseExperiment[ResultBundle]` when it stores no bundles)
- Public APIs expose explicit named parameters rather than bare `*args` / `**kwargs`; use keyword-only optional controls for public plotting and plot-data APIs (enforced by `causalpy/tests/test_public_signatures.py` and surveyed by `scripts/audit_public_signatures.py`). A genuine dynamic or third-party forwarder requires an `Other Parameters` contract and a narrow structural-test exemption. For experiments without a unified plot view (e.g. `InversePropensityWeighting`, `InstrumentalVariable`), declare an explicit `plot()` stub that raises `NotImplementedError`. For `hdi_prob` defaults, use ``Defaults to :data:`~causalpy.constants.HDI_PROB` (currently 0.94).`` in the docstring.
Comment thread
anevolbap marked this conversation as resolved.
- Raise `FormulaException`, `DataException`, or `BadIndexException` from `causalpy.custom_exceptions` for formula, data, and index errors
- Avoid backwards-compat shims for APIs introduced in the same PR
Expand Down
12 changes: 12 additions & 0 deletions causalpy/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,13 @@
from causalpy.transforms import ramp, step
from causalpy.version import __version__

from .custom_exceptions import (
BadIndexException,
DataException,
FormulaException,
GroupNotSampledException,
PriorPredictiveNotSupportedException,
)
from .data import load_data
from .experiments.diff_in_diff import DifferenceInDifferences
from .experiments.instrumental_variable import InstrumentalVariable
Expand Down Expand Up @@ -49,13 +56,17 @@

__all__ = [
"__version__",
"BadIndexException",
"checks",
"DataException",
"create_causalpy_compatible_class",
"DifferenceInDifferences",
"EstimateEffect",
"EffectSummary",
"extract_lift_for_mmm",
"FormulaException",
"GenerateReport",
"GroupNotSampledException",
"InstrumentalVariable",
"InterruptedTimeSeries",
"InversePropensityWeighting",
Expand All @@ -67,6 +78,7 @@
"PanelRegression",
"plot_correlations",
"PrePostNEGD",
"PriorPredictiveNotSupportedException",
"pymc_forecast_models",
"pymc_models",
"ramp",
Expand Down
2 changes: 1 addition & 1 deletion causalpy/checks/bandwidth.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ def run(
kw["model"] = clone_model(kw["model"])

try:
alt_experiment = method(context.data, **kw)
alt_experiment = method(context.data, **kw).fit()
summary = alt_experiment.effect_summary()
row: dict[str, Any] = {"bandwidth": bw}
if summary.table is not None and not summary.table.empty:
Expand Down
2 changes: 1 addition & 1 deletion causalpy/checks/leave_one_out.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ def run(
kw["model"] = clone_model(kw["model"])

try:
alt_experiment = method(context.data, **kw)
alt_experiment = method(context.data, **kw).fit()
summary = alt_experiment.effect_summary()
row: dict[str, Any] = {"dropped_unit": dropped}
if summary.table is not None and not summary.table.empty:
Expand Down
2 changes: 1 addition & 1 deletion causalpy/checks/outcome_falsification.py
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ def _build_experiment(
if "model" in kwargs and kwargs["model"] is not None:
kwargs["model"] = clone_model(kwargs["model"])

return method(context.data, **kwargs)
return method(context.data, **kwargs).fit()

@staticmethod
def _extract_effect_stats(
Expand Down
2 changes: 1 addition & 1 deletion causalpy/checks/placebo_in_space.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ def run(
kw["model"] = clone_model(kw["model"])

try:
alt_experiment = method(context.data, **kw)
alt_experiment = method(context.data, **kw).fit()
summary = alt_experiment.effect_summary()
row: dict[str, Any] = {"placebo_treated": placebo_treated}
if summary.table is not None and not summary.table.empty:
Expand Down
6 changes: 3 additions & 3 deletions causalpy/checks/placebo_in_time.py
Original file line number Diff line number Diff line change
Expand Up @@ -439,7 +439,7 @@ def validate(self, experiment: BaseExperiment) -> None:
f"explicit treatment time."
)
# Any InferenceData-capable backend (PyMCModel or PyMCForecastModel)
# yields the draw-level post_impact this check consumes.
# yields the draw-level posterior impact this check consumes.
backend = getattr(experiment, "_model_backend", None)
if backend is None or not backend.supports_idata:
raise TypeError(
Expand Down Expand Up @@ -518,7 +518,7 @@ def _factory(
kw["model"] = self._clone_model_for_fold(
model_template, fold_random_seed
)
return method(data, **kw)
return method(data, **kw).fit()

return _factory

Expand Down Expand Up @@ -835,7 +835,7 @@ def _extract_cumulative_impact(experiment: BaseExperiment) -> xr.DataArray:
obtained by summing over ``obs_ind`` and stacking
``(chain, draw)``.
"""
post_impact = experiment.post_impact # type: ignore[attr-defined]
post_impact = experiment.result.impact_post

if "treated_units" in post_impact.dims:
post_impact = post_impact.isel(treated_units=0)
Expand Down
11 changes: 6 additions & 5 deletions causalpy/checks/pre_treatment_placebo.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ class PreTreatmentPlaceboCheck:
"""Check that pre-treatment event-study estimates are near zero.

Wraps the pre-treatment placebo effects already computed by
``StaggeredDifferenceInDifferences`` in ``att_event_time_``.
``StaggeredDifferenceInDifferences`` in its result bundle
(``result.att_event_time``).

Parameters
----------
Expand Down Expand Up @@ -61,10 +62,10 @@ def validate(self, experiment: BaseExperiment) -> None:
"PreTreatmentPlaceboCheck requires a "
"StaggeredDifferenceInDifferences experiment."
)
if not hasattr(experiment, "att_event_time_"):
if not experiment.is_fitted:
raise ValueError(
"Experiment does not have att_event_time_. "
"Ensure the experiment has been fitted."
"Experiment has not been fitted. "
"Ensure fit() has been called before running this check."
)

def run(
Expand All @@ -82,7 +83,7 @@ def run(
Pipeline context (unused; required by the check protocol).
"""
sdid = experiment
att_et = sdid.att_event_time_ # type: ignore[attr-defined]
att_et = sdid.result.att_event_time

pre_treatment = att_et[att_et["event_time"] < 0].copy()

Expand Down
2 changes: 1 addition & 1 deletion causalpy/checks/prior_sensitivity.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@ def run(
model = clone_model(alt["model"])
logger.info("PriorSensitivity: fitting with '%s'", name)

alt_experiment = method(context.data, model=model, **base_kwargs)
alt_experiment = method(context.data, model=model, **base_kwargs).fit()

try:
summary = alt_experiment.effect_summary()
Expand Down
43 changes: 43 additions & 0 deletions causalpy/custom_exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,3 +58,46 @@ class DataException(Exception):
def __init__(self, message: str):
super().__init__(message)
self.message = message


class GroupNotSampledException(Exception):
"""Raised when a read method requests a draw group that has not been sampled.

Under the lazy lifecycle an experiment holds no draws until
:meth:`~causalpy.experiments.base.BaseExperiment.fit` (posterior group) or
:meth:`~causalpy.experiments.base.BaseExperiment.sample_prior_predictive`
(prior group) is called. This exception carries the missing group and the
call that would populate it so the fix is actionable.

Parameters
----------
message : str
Human-readable description naming the missing group and the call to
make.
group : str, optional
The draw group that was requested but not sampled.
"""

def __init__(self, message: str, group: str | None = None):
super().__init__(message)
self.message = message
self.group = group


class PriorPredictiveNotSupportedException(Exception):
"""Raised when prior predictive sampling is requested of a backend that cannot do it.

Prior-phase support is a property of the model backend. OLS/sklearn
models, ``PyMCForecastModel``, and the PyMC state-space and instrumental
variable models do not expose a prior predictive phase.

Parameters
----------
message : str
Human-readable description naming the model class that lacks the
capability.
"""

def __init__(self, message: str):
super().__init__(message)
self.message = message
150 changes: 150 additions & 0 deletions causalpy/experiments/_results.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# Copyright 2022 - 2026 The PyMC Labs Developers
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Per-experiment result bundles for the lazy experiment lifecycle.

Every bundle is *fully populated* whenever it exists: an experiment either has
no bundle yet (the corresponding lifecycle verb has not run) or a complete one.
Nothing derived from model draws lives directly on the experiment object; the
two public slots ``experiment.result`` (posterior group) and
``experiment.prior_result`` (prior group) hold these bundles and back the
``is_fitted`` / ``has_prior_predictive`` state predicates.

All prediction/impact fields are typed :class:`xarray.DataArray` on the
canonical ``("chain", "draw", "obs_ind"[, ...])`` dimensions produced by
:class:`~causalpy.experiments.model_adapter.ModelAdapter.predict`.
"""

from __future__ import annotations

from dataclasses import dataclass, field

import pandas as pd
import xarray as xr

__all__ = [
"CausalResult",
"CoefficientResult",
"DiscontinuityResult",
"GroupComparisonScenario",
"KinkResult",
"ResultBundle",
"StaggeredDifferenceInDifferencesResult",
"SyntheticDifferenceInDifferencesResult",
]


@dataclass(frozen=True)
class CausalResult:
"""Result bundle for predict-contrast experiments.

Used by :class:`~causalpy.experiments.interrupted_time_series.InterruptedTimeSeries`,
:class:`~causalpy.experiments.synthetic_control.SyntheticControl`, and
:class:`~causalpy.experiments.piecewise_its.PiecewiseITS`.

For ``PiecewiseITS`` ``predictions_pre`` carries the fitted expectation over
the full observation window and ``predictions_post`` / ``impact_post`` /
``impact_post_cumulative`` carry the post-first-interruption slices consumed
by the reporting helpers.
"""

predictions_pre: xr.DataArray
predictions_post: xr.DataArray
impact_pre: xr.DataArray
impact_post: xr.DataArray
impact_post_cumulative: xr.DataArray
score: pd.Series | None = None


@dataclass(frozen=True)
class SyntheticDifferenceInDifferencesResult(CausalResult):
"""Result bundle for :class:`~causalpy.experiments.synthetic_difference_in_differences.SyntheticDifferenceInDifferences`.

The ``CausalResult`` prediction/impact fields are reconstructed from the
synthetic-control imputation; ``tau_posterior`` carries the analytic
double-difference treatment-effect draws with dimensions
``("chain", "draw")``.
"""

tau_posterior: xr.DataArray = field(kw_only=True)


@dataclass(frozen=True)
class GroupComparisonScenario:
"""One scenario plotted or summarized by a group-comparison experiment."""

inputs: pd.DataFrame
prediction: xr.DataArray


@dataclass(frozen=True)
class CoefficientResult:
"""Result bundle for coefficient-contrast experiments (DiD, PrePostNEGD).

``causal_impact`` holds draws of the treatment-effect coefficient (or its
algebraically-equivalent prediction contrast) with canonical coefficient
dimensions.
"""

causal_impact: xr.DataArray
scenario_control: GroupComparisonScenario
scenario_treated: GroupComparisonScenario
scenario_counterfactual: GroupComparisonScenario
score: pd.Series | None = None


@dataclass(frozen=True)
class DiscontinuityResult:
"""Result bundle for :class:`~causalpy.experiments.regression_discontinuity.RegressionDiscontinuity`."""

predictions: xr.DataArray
discontinuity_at_threshold: xr.DataArray
score: pd.Series | None = None


@dataclass(frozen=True)
class KinkResult:
"""Result bundle for :class:`~causalpy.experiments.regression_kink.RegressionKink`."""

predictions: xr.DataArray
gradient_change: xr.DataArray
score: pd.Series | None = None


@dataclass(frozen=True)
class StaggeredDifferenceInDifferencesResult:
"""Result bundle for :class:`~causalpy.experiments.staggered_did.StaggeredDifferenceInDifferences`.

``att_group_time`` and ``att_event_time`` are aggregated ATT tables;
``y_pred`` retains the raw counterfactual draws so placebos and alternate
HDI levels can re-derive effects without resampling.
"""

att_group_time: pd.DataFrame
att_event_time: pd.DataFrame
y_pred: xr.DataArray
hdi_prob: float
score: pd.Series | None = None


#: Every bundle type the raising ``result`` / ``prior_result`` properties may
#: return, one per experiment family. Experiments without bundles (IV, IPW,
#: PanelRegression) never produce these.
ResultBundle = (
CausalResult
| CoefficientResult
| DiscontinuityResult
| KinkResult
| StaggeredDifferenceInDifferencesResult
| SyntheticDifferenceInDifferencesResult
)
Loading