Don't leak __tmp_weights column out of groupby - #286
Merged
Conversation
MaxGhenis
commented
Apr 17, 2026
MaxGhenis
left a comment
Collaborator
Author
There was a problem hiding this comment.
Self-review. Verified:
- Old path
self["__tmp_weights"] = self.weightsmutated caller; new path copiespd.DataFrame(self).copy()intostagedand adds the weight column there.self.columnsis untouched. gb = staged.groupby(...); subsequentgb["__tmp_weights"]still resolves because the column lives onstaged(which the GroupBy object holds).- Loop
for col in staged.columnsiterates overself.columns + __tmp_weights; per-columnMicroSeriesGroupByrewrapping is unchanged. - CoW note (pandas 3):
pd.DataFrame(self).copy()returns a detached copy, so even under CoW the staging column mutation doesn't reachself. Safe. - Regression test verifies single-column and list-column groupby don't leak
__tmp_weights, and that the weighted sum result is still correct. - CI green Py 3.9-3.13.
MicroDataFrame.groupby used to do self["__tmp_weights"] = self.weights
and never clean it up, so after a single df.groupby(...) call the
caller's df permanently carried the weight column:
df = MicroDataFrame({"g":["a","a","b"], "v":[1,2,3]}, weights=[1,2,3])
_ = df.groupby("g").sum()
list(df.columns) # ['g', 'v', '__tmp_weights']
Any later df.sum() or iteration over columns then picked it up as data.
Fix: stage the weights onto a copy before calling super().groupby()
rather than mutating self. All existing groupby plumbing
(_weights_groupby, per-column MicroSeriesGroupBy) keeps working.
MaxGhenis
force-pushed
the
fix/groupby-tmp-weights-leak
branch
from
April 17, 2026 16:15
81b4056 to
3a1fca7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MicroDataFrame.groupbyused to doself["__tmp_weights"] = self.weightsand never clean it up, so after a singledf.groupby(...)call the caller's DataFrame permanently carried the weight column. Any laterdf.sum()or iteration over columns then picked it up as data.Fix: stage the weights onto a copy before calling
super().groupby()rather than mutatingself. All existing groupby plumbing (_weights_groupby, per-columnMicroSeriesGroupBy) keeps working.Reproduction
Test plan
test_groupby_does_not_leak_tmp_weights_columncovers single-column and list-of-columns groupbys and verifies the weighted aggregation result stays correct.