Skip to content

BUG: Return the contribution of the absorbed terms in absorbed_effects - #786

Merged
bashtage merged 1 commit into
mainfrom
lm71-fix-absorbed-effects
Oct 12, 2026
Merged

bashtage merged 1 commit into
mainfrom
lm71-fix-absorbed-effects

Conversation

@bashtage

Copy link
Copy Markdown
Owner

AbsorbingLSResults.absorbed_effects is documented as the fitted values from only the absorbed terms, but it was the purged dependent variable minus the fitted values. That is the dependent variable after the effects were removed (scaled by the square root of the weights) less the fitted values of the whole model, which has one value for each observation and is no quantity of use to a user. With one categorical variable of 12 categories and 600 observations it had 600 different values, where the contribution of the effects has 12, and it differed from the right values by up to 7.5. fitted_values and resids were correct.

absorbed_effects is now fitted_values - exog @ params, in the units of the dependent variable, where exog holds the regressors that were estimated. It is the contribution of the effects, the interactions and the continuous variables that were absorbed, and of any regressor that was dropped because the effects absorbed it. It equals the part of the fitted values that the dummy variables of the effects explain in the regression on the regressors and the dummy variables, with or without weights, with interactions and continuous variables, and with rows dropped because of missing values. If exog contains a constant and the effects include a categorical variable, the constant has the mean of the effects, so they have mean 0 (weighted). The docstring says so.

AbsorbingLSResults.absorbed_effects is documented as the fitted values from only the
absorbed terms, but it was the purged dependent variable minus the fitted values.
That is the dependent variable after the effects were removed (scaled by the square
root of the weights) less the fitted values of the whole model, which has one value
for each observation and is no quantity of use to a user. With one categorical
variable of 12 categories and 600 observations it had 600 different values, where
the contribution of the effects has 12, and it differed from the right values by up
to 7.5. fitted_values and resids were correct.

absorbed_effects is now fitted_values - exog @ params, in the units of the
dependent variable, where exog holds the regressors that were estimated. It is the
contribution of the effects, the interactions and the continuous variables that were
absorbed, and of any regressor that was dropped because the effects absorbed it. It
equals the part of the fitted values that the dummy variables of the effects explain
in the regression on the regressors and the dummy variables, with or without
weights, with interactions and continuous variables, and with rows dropped because
of missing values. If exog contains a constant and the effects include a categorical
variable, the constant has the mean of the effects, so they have mean 0 (weighted).
The docstring says so.
@codecov

codecov Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.59%. Comparing base (121d869) to head (82a0f51).
⚠️ Report is 77 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #786      +/-   ##
==========================================
- Coverage   99.60%   99.59%   -0.01%     
==========================================
  Files         127      129       +2     
  Lines       23832    24569     +737     
  Branches     1783     1821      +38     
==========================================
+ Hits        23737    24470     +733     
- Misses         35       39       +4     
  Partials       60       60              
Flag Coverage Δ
adder 99.58% <100.00%> (-0.01%) ⬇️
subtractor 99.58% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bashtage
bashtage merged commit 06d4713 into main Oct 12, 2026
24 checks passed
@bashtage
bashtage deleted the lm71-fix-absorbed-effects branch October 12, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant