Skip to content

GH-4242: Do not rewrite DISTINCT to REDUCED when ORDER BY is on an expression - #4245

Open
anishmehta24 wants to merge 1 commit into
apache:mainfrom
anishmehta24:gh-4242-distinct-reduced-expr-order
Open

anishmehta24 wants to merge 1 commit into
apache:mainfrom
anishmehta24:gh-4242-distinct-reduced-expr-order

Conversation

@anishmehta24

Copy link
Copy Markdown

GitHub issue resolved #4242

Pull request Description:

TransformDistinctToReduced rewrites DISTINCT + ORDER BY to REDUCED + ORDER BY when the ordering makes equal projected rows adjacent. Its isValidSortCondition treated an ORDER BY expression like a plain variable: every variable mentioned by the expression was added to seenVars. Ordering by STRLEN(?v) does not make equal values of ?v adjacent (the function is not injective), so with ORDER BY STRLEN(?v) ?tag and ?tag unprojected, the rows ("a",1) ("b",1) ("a",2) sort to a b a and REDUCED returns "a" twice.

The change keeps allowing expressions over projected variables in the ordering (they do not break the property) but they no longer count towards "all projected variables seen"; only a projected variable ordered on directly does. So:

  • ORDER BY STRLEN(?v) ?tag and ORDER BY STRLEN(?v) stay DISTINCT (new distinct_to_reduced_14/15, both fail on main).
  • ORDER BY STRLEN(?v) ?v ?tag is still rewritten to REDUCED (new distinct_to_reduced_16).
  • Scripted query from the issue in testing/ARQ/Extra (distinct-order-expr), which fails on main with the duplicate row.

TestOptDistinctReduced, TestOptimizer and Scripts_ARQ pass. TransformOrderByDistinctApplication was not touched: it only considers plain variables already.


  • Tests are included.
  • Documentation change and updates are provided for the Apache Jena website (not needed)
  • Commits have been squashed to remove intermediate development commit messages.
  • Key commit messages start with the issue number (GH-xxxx)

By submitting this pull request, I acknowledge that I am making a contribution to the Apache Software Foundation under the terms and conditions of the Contributor's Agreement.

🤖 Generated with Claude Code

… an expression

TransformDistinctToReduced counted the variables of an ORDER BY
expression as seen, as if ordering by STRLEN(?v) made equal values of ?v
adjacent. It does not unless the expression is injective, so with
ORDER BY STRLEN(?v) ?tag and ?tag not projected, REDUCED let duplicate
?v rows through. An expression over projected variables is still allowed
in the ordering but no longer counts towards the projected variables
having been seen.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DISTINCT with ORDER BY on a non-injective expression is rewritten to REDUCED and leaks duplicate rows

1 participant