Filter and sort DataFrames without a row index - #8727
Draft
ghostiee-11 wants to merge 14 commits into
Draft
Conversation
Panel already depends on narwhals and Bokeh already accepts any narwhals compatible frame as ColumnDataSource input, but Panel's own tabular components remain pandas only. panel/_dataframe.py becomes the single place that answers "is this a DataFrame" and "give me a common API over it" for any backend, so the migration has one seam to move rather than call sites scattered across tables.py, reactive.py and perspective.py. Nothing imports it yet, so there is no behaviour change. narwhals is imported inside the functions, matching panel/pane/vega.py, so `import panel` does not pay its ~40ms. panel.util.checks.is_dataframe is deliberately left alone. Its callers go on to use the pandas API, such as calling .all() on the result of a frame comparison in pane/base.py, so widening it there would change behaviour.
Covers the new seam across pandas, Polars and PyArrow, and pins the current state of the four tabular entry points. The Polars and PyArrow cases are strict xfails, so the change that makes one of them work fails the suite until the expectation is updated. That is what lets the rest of this migration land in small steps without silently regressing or silently succeeding. polars and pyarrow are importorskip'd, since neither is in the test-core environment.
pandas and the libraries that copy its API carry a row index that can label rows. Polars and PyArrow do not. Code that addresses a row by label needs to know which it is holding so it can fall back to addressing by position.
Tabulator and DataFrame can now be constructed from Polars and PyArrow frames. Six places assumed a pandas row index or pandas column labels: - indexes returns nothing to show when the frame has no index - _validate reads column names through narwhals, since PyArrow's .columns is the column data rather than the names - _get_fields likewise - _index_mapping is the identity when position is all a row has - _process_df_and_convert_to_cds hands the frame straight to Bokeh, which already serializes any narwhals compatible frame, since there is no index to flatten into columns first - _update_selected treats a selected row as its own position The pandas paths are untouched; every branch added is behind a has_index() check, and the existing table suite passes unmodified. Rendering these frames still needs backend neutral dtype handling in _get_column_definitions, so that case stays a strict xfail.
Not re-exported from panel.util, since util already exports a pandas specific is_dataframe from checks and the two names would collide. Importing from panel.util.dataframe keeps the distinction visible.
Panel picks editors, formatters and sorters off numpy's single letter dtype kinds. narwhals describes types with predicates instead, so dtype_kind bridges the two and only emits the kinds Panel branches on.
Column definitions and the Tabulator column config both read dtypes off numpy, which Polars and PyArrow do not provide. Both now go through the narwhals schema when the frame has no index, leaving the pandas paths on their original code so kinds numpy distinguishes and narwhals does not, such as unsigned integers, keep behaving as before. Date formatting needed care: narwhals returns real datetimes and datetime subclasses date, so a datetime column would otherwise be formatted as a bare date. pandas and Polars now produce the same format for both date and datetime columns. Tabulator and DataFrame now render Polars and PyArrow frames, so those xfails are gone. The DataFrame and Perspective panes still do not.
Adding the narwhals branch to _get_column_definitions pushed the loop body to four levels of nesting and left the `pd.Series | pd.Index` annotation on a variable that can now also hold a narwhals Series. Moving the lookup into its own method flattens the loop back to one line and lets each branch return the pair it actually produces.
Header filters and add_filter now work on Polars and PyArrow. The pandas implementation builds boolean Series and selects with df[mask]; narwhals builds expressions and selects with df.filter(expr), so the index-less path is a parallel implementation rather than a shared abstraction. Unifying them would put years of pandas filter behaviour at risk for no user visible gain, and the backend parametrised tests check that the two agree on results. Two details worth noting: Tabulator sends every header filter value as a string, so numeric columns need the value cast to the column type or the comparison becomes a string comparison. pandas got this for free from col.dtype.type(val). Only real frames are diverted. Tabulator.value can also be None or a dict, and those keep going through the original code. Before this, filters on a PyArrow table silently returned every row rather than raising, which is the worse of the two failure modes.
Reading column names through narwhals for every backend meant a pandas frame with duplicate columns hit narwhals' DuplicateError before _validate could report it. That error subclasses ValueError so the existing test still passed, but the message went from Panel telling the user what was wrong to a generic one from a library they did not call. column_names() now reads pandas directly and only routes other backends through narwhals, which is where the PyArrow .columns problem actually is.
…(cherry-picked from holoviz#8726)
The pandas implementation sorts with kind='mergesort' and a key= callable that lowercases string columns, so results match Tabulator's own case insensitive ordering. Narwhals has no sort(key=), so the same result is built explicitly: a derived lowercase column per string sort key, and a row index appended as the final key to break ties stably. Verified to produce identical ordering to pandas for ascending, descending, string, numeric and multi key sorts.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #8727 +/- ##
========================================
Coverage 85.41% 85.41%
========================================
Files 350 352 +2
Lines 57604 57921 +317
========================================
+ Hits 49202 49476 +274
- Misses 8402 8445 +43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #8723, which adds the seam and makes the table widgets construct and render Polars and PyArrow frames. This adds the rest of the read path: dtypes drive editors and formatters, and filtering and sorting work on any backend.
The narwhals paths are written alongside the pandas ones rather than replacing them, since the two build masks and sort keys in different ways, and the parametrised tests check they agree. Part of #8721.
Draft because it stacks on #8723 and #8726, so the first commits here are theirs. I will rebase once those land.