Repository navigation
ENH: Add extra_df to the IV covariance estimators - #778
Merged
Merged
Conversation
The small-sample adjustment of the unadjusted, robust, kernel and clustered covariance estimators is n / (n - k), where k is the number of columns of x. extra_df is the number of degrees of freedom that the model uses in addition to them, for example the absorbed effects of AbsorbingLS. If debiased is True the adjustment is n / (n - k - extra_df), and extra_df is not used if debiased is False. It is 0 by default, which is the previous adjustment, and the covariance estimators of the panel models have the same argument. A positive extra_df that leaves no degrees of freedom raises a ValueError that says so, instead of a division by zero or a negative scale.
…bingLS AbsorbingLS(debiased=True) scaled the covariance by n / (n - k), where k counts the regressors of exog and not the absorbed terms, although df_resid and the t and F tests count them. The standard errors were too small compared with the regression on the regressors and the dummy variables of the effects and with PanelOLS(debiased=True), which agree. With the Grunfeld data of #432 (n = 220, the 2 regressors value and capital, 11 firm and 20 year effects, so 30 absorbed degrees of freedom) the unadjusted standard errors were 0.012010 and 0.019547 where the regression has 0.012933 and 0.021049, and the robust standard errors 0.017738 and 0.049098 where it has 0.019101 and 0.052870. The ratio is sqrt(188 / 218) = 0.9286, since the variance was scaled by n / (n - 2) = 1.009174 and not by n / (n - 32) = 1.170213 (#220, #432). The small-sample adjustment now counts the degrees of freedom of the absorbed terms: the dummy variables of the effects (one fewer if exog has a constant, and one fewer for each effect after the first), the absorbed continuous variables and the interactions, which is df_absorbed. The unadjusted, robust, kernel and clustered covariances are equal to those of the regression on the regressors and the dummy variables of the effects with debiased=True, in simulations with up to 3 effects, weights, interactions, continuous variables and a constant, and the standard errors are equal to those of PanelOLS(debiased=True) for the unadjusted and robust covariances, and for the clustered covariance with group_debias=True. The exception is the clustered covariance of a model with a single categorical effect that is nested in every cluster variable: its degrees of freedom are not counted, as in PanelOLS and in Stata. Absorbed continuous variables and interactions are still counted. With two or more effects all of the effects are counted, as in PanelOLS. Nothing changes if debiased is False, where no adjustment is applied: unlike PanelOLS, which always adjusts for its effects. The change makes the standard errors of debiased=True larger, by a factor that is close to 1 if n is large relative to the number of absorbed terms. A debiased covariance of a model that uses all of its degrees of freedom raises a ValueError. fit documents the adjustment in its Notes.
… as it was The Notes of fit explain what debiased does for the absorbed terms, so the entry in Parameters does not need to change. Leaving its lines alone keeps the change separate from the new text of the method entry, which another change to the same docstring edits on the lines that follow it.
…ue to match The covariance of AbsorbingLS with debiased=True equals that of PanelOLS(debiased=True) with the same effects for the unadjusted and robust estimators. For the clustered covariance PanelOLS has to use group_debias=True, which applies the adjustment for the number of clusters that AbsorbingLS always applies if debiased is True.
…degrees of freedom fit raises a ValueError if debiased is True and the regressors and the absorbed terms use all of the observations. The Notes say so.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #778 +/- ##
==========================================
+ Coverage 99.58% 99.59% +0.01%
==========================================
Files 128 129 +1
Lines 24188 24693 +505
Branches 1809 1827 +18
==========================================
+ Hits 24087 24594 +507
+ Misses 40 39 -1
+ Partials 61 60 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The small-sample adjustment of the unadjusted, robust, kernel and clustered
covariance estimators is n / (n - k), where k is the number of columns of x.
extra_df is the number of degrees of freedom that the model uses in addition to
them, for example the absorbed effects of AbsorbingLS. If debiased is True the
adjustment is n / (n - k - extra_df), and extra_df is not used if debiased is
False. It is 0 by default, which is the previous adjustment, and the covariance
estimators of the panel models have the same argument. A positive extra_df that
leaves no degrees of freedom raises a ValueError that says so, instead of a
division by zero or a negative scale.