Repository navigation
Replace lets by their values in lambdas handed to a LINQ provider - #20895
xperiandri wants to merge 13 commits into
Conversation
Every query test so far runs over AsQueryable() and compares rows, so a tree that only LINQ to Objects can run passes. These scenarios also check the tree: the query must stay on its provider, and must not contain what a C# expression lambda never does - a block, a parameter nothing binds, a call into FSharp.Core, a join nested in a collection selector, a member read from a tuple built without naming its members. A scenario names what is wrong with its tree today. The name has to be removed by the change that makes it untrue, because a scenario fails when its tree is cleaner than it says. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Selecting a struct tuple failed with "unreachable": the tuple was replaced by an anonymous object, but only reference tuples had an anonymous object type to be replaced by, and the way back always built a reference tuple. Both kinds now share the anonymous object of their arity, and the way back builds the kind the query asked for. Taking a struct tuple apart in a lambda failed with a tuple index out of range, because a struct tuple has fields where a reference tuple has properties. Fixes dotnet#15522 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
✅ Release notes checked
|
Covered casesEF Core SQLite translates 122 of 145 scenarios, 92 without this change. Now translated: records and anonymous records with fields out of order in Cosmos translates 26 scenarios, up from 17. linq2db returned wrong rows for Not changed: a call that is not a member of a string, number or date keeps its block when a |
There was a problem hiding this comment.
🟡 Changes recommended
Critical substitution issues can reorder explicit-let evaluation, mutate struct fields instead of local copies, and duplicate side effects through provider callbacks or custom nullable structs.
4 open issues
What changed in this PR
This PR improves FSharp.Core’s LINQ-provider compatibility by replacing eligible let bindings with their values instead of emitting Expression.Block. It builds on #20894’s struct-tuple support and query-shape tests to address record-field ordering, tupled parameters, and nullable-property access.
Changes:
- Substitute eligible bindings inside provider expression lambdas while retaining blocks for guarded cases.
- Add query-shape and evaluation-semantics regression coverage.
- Document expression-tree changes and their evaluation-order impact.
| File | Description |
|---|---|
| tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Linq/QueryStructureScenarios.fs | Covers loops, subqueries, and left joins. |
| tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Linq/QueryShapeTests.fs | Runs the query scenario suites. |
| tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Linq/QueryScenarioDomain.fs | Supplies shared query data and types. |
| tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Linq/QueryScenario.fs | Checks results, provider retention, and tree shape. |
| tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Linq/QueryProviderShape.fs | Captures queries and detects unsupported tree patterns. |
| tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Linq/QueryProjectionScenarios.fs | Covers tuple and record projections. |
| tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Linq/QueryOperatorScenarios.fs | Covers filtering, ordering, and aggregates. |
| tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Linq/QueryMethodSyntaxScenarios.fs | Covers direct LINQ method calls. |
| tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Linq/QueryKeyScenarios.fs | Covers composite join and grouping keys. |
| tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Linq/ExpressionLambdaConversionTests.fs | Tests substitution, evaluation order, and identity. |
| tests/FSharp.Core.UnitTests/FSharp.Core.UnitTests.fsproj | Registers the new test files. |
| src/FSharp.Core/QueryExtensions.fs | Adapts struct tuples to query representations. |
| src/FSharp.Core/Query.fs | Integrates provider-specific conversion and record-field inlining. |
| src/FSharp.Core/Linq.fsi | Declares internal conversion helpers. |
| src/FSharp.Core/Linq.fs | Implements binding substitution and struct-tuple access. |
| docs/release-notes/PENDING_BREAKING_CHANGES.md | Documents tree-shape and evaluation-order changes. |
| docs/release-notes/.FSharp.Core/11.0.200.md | Adds LINQ compatibility release notes. |
Review effort: Balanced
The shape tests followed only the first source of each operator. Each lambda parameter is now tied to the sequence it stands for, so the check also covers the inner input of a join, the second parameter of a join result selector, a collection selector of SelectMany, an identity Select, a chain inside the lambda of another operator and a query captured from earlier. A value scenario now has to end in a Queryable operator that gives the value: a count made over the rows of the provider leaves it. The validator itself is tested on trees built by hand. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ac05f7d to
31c93a1
Compare
pcshrosbree
left a comment
There was a problem hiding this comment.
At the current head, the #11332 predicate gives the same result as plain F#, and its expression tree contains no blocks. Please expand PENDING_BREAKING_CHANGES and add tests for the remaining changes in when and how often expressions run. I ran the expression trees through AsQueryable() and compared the results with main and plain F#.
In the record-field cases I checked, user-written let bindings to Counter.Next() now keep their block unless the bound variable has the same name as the field that uses it. The example below shows that exception: the call bound to B still runs after the call for A.
let B = Counter.Next() in { A = Counter.Next(); B = B }
// This PR: { A = 1; B = 2 }
// Plain F#: { A = 2; B = 1 }The note could mention this remaining reordering.
PENDING_BREAKING_CHANGES:
- A member read substituted into a nested
Funclambda can run once per inner element. WithnamesaList<string>,custs.Select(fun c -> let n = c.Name in names.Count(fun s -> s = n))readsName8 times for 2 customers and 4 names; main reads it twice. This also affects arithmetic expressions bound bylet, such aslet a = x.Age + 1, when a nested lambda uses the bound value. In that probe,Agewas read 4 times, compared with once on main. - The note says
EvaluateQuotationandQuotationToExpressionare unaffected. ButQuotationToLambdaExpressionchanges for every caller, andEvaluateQuotation <@ ppl.Select(fun x -> let n = x.Name in n + n).ToArray() @>readsNametwice where main reads it once. - A
letbinding that would throw can now be skipped when a conditional takes a branch that does not use the bound value.let s = x.Name.Substring(5) in if x.Name.Length > 5 then s else ""returns""for"ab", where F# throwsArgumentOutOfRangeException.
Tests. Please add tests that count reads of a getter on the lambda's parameter, including a case where a nested lambda uses the getter's bound value. "Member of an object read before a call keeps the value it had" passes because items is a captured constant. If items is replaced with the lambda's parameter, the result is 1 on main and 2 on this PR.
The join-key fallback in Query.fs also fixes join p in ps on ((k.KA, k.KB) = p.Pair), which throws Type mismatch when building 'ty' on main; it could have a release-note line and a test.
T-Gro
left a comment
There was a problem hiding this comment.
🤖🕵️ AI review — verify independently.
| let isPlainValueMethod (call: MethodCallExpression) = | ||
| match call.Object with | ||
| | null -> false | ||
| | _ -> isPlainValueType call.Method.DeclaringType && isPlainValueType call.Type |
There was a problem hiding this comment.
🤖🕵️ [P2] Lost string identity after duplicating a call, even without getters or side effects.
open System
open Microsoft.FSharp.Linq.RuntimeHelpers
let f = LeafExpressionConverter.QuotationToLambdaExpression
<@ Func<string, bool>(fun s ->
let upper = s.ToUpperInvariant()
obj.ReferenceEquals(upper, upper)) @>
f.Compile().Invoke "ab" // base: true; this head: falseThere was a problem hiding this comment.
Fixed in 4360ae3. A let whose value a call makes anew is no longer replaced at each of several uses when one of them can tell one object from another: the visitor ObjectIdentityUses looks at what the rest of the lambda does with the variable, so obj.ReferenceEquals(upper, upper) keeps its block and is true. Uses that only look at the string, like upper.StartsWith("A") && upper.EndsWith("B"), still take the value in. The tests are String made by a call is one string where a use can tell one from another and String made by a call is made at each use that only looks at it.
There was a problem hiding this comment.
🤖🕵️ Still loses string identity at 9e58c229 through an identity-preserving call or a single captured use in a nested lambda.
open System
open System.Linq
open Microsoft.FSharp.Linq.RuntimeHelpers
let f = LeafExpressionConverter.QuotationToLambdaExpression
<@ Func<string, bool>(fun s ->
let upper = s.ToUpperInvariant()
obj.ReferenceEquals(upper.Substring(0), upper.Substring(0))) @>
f.Compile().Invoke "ab" // base: true; 9e58c229: false
let g = LeafExpressionConverter.QuotationToLambdaExpression
<@ Func<string, Collections.Generic.IEnumerable<string>>(fun s ->
let upper = s.ToUpperInvariant()
Enumerable.Select([| 1; 2 |], Func<int, string>(fun _ -> upper))) @>
let items = g.Compile().Invoke("ab").ToArray()
obj.ReferenceEquals(items.[0], items.[1]) // base: true; 9e58c229: falseA value scenario was accepted when the last execution at the provider was an operator that gives a value. A count made by the client over rows taken from the provider passed whenever a function it ran asked the provider for a count of its own, because that count came last. Now every execution has to be such an operator, and so does the enumeration of a source taken as it is, which is recorded as an execution too. The violations of a tree are compared with every occurrence: a tree with one block was equal to a tree with two. The scenarios that have several of a kind are pinned with their number. The validator tests build a tree with an invoked lambda and one with an unbound parameter, which no scenario has, and the two client counts. The release note names the error a struct tuple gave on main. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…y what they hold The shape tests did not see an unnamed element that passes through a result selector giving one of its parameters back, in SelectMany, Join and Zip, and they skipped the element parameter of Aggregate together with the accumulated value. Both are checked now. A function with a seed takes the accumulated value and an element, and one without a seed takes two elements. A scenario compared an entity by the text it makes of itself, which names its key only: a result with the right key and the wrong name or age passed. An object of a class of the scenarios is now compared by every property that gives a plain value. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A record or anonymous record with fields written out of order, a tupled lambda parameter, a struct copy such as x.Nullable.HasValue on a class property, or an explicit let: each of these is a let in the quotation, and the converter turned it into an Expression.Block with a local variable. No LINQ provider translates a block, so EF Core and Cosmos failed on all of these. Inside a lambda that goes to a provider - a query operator over IQueryable, or an F# lambda passed as Expression<Func<..>> - the value now takes the place of the variable where that computes the same: - the value only reads: a constant, a parameter of the lambda or a member of one, or what a member or method of a string, number or date makes of those. Such a read can be made at every use; - or the variable is read once, as the first thing the rest evaluates, so that the value is computed at the same moment as before. Any other let keeps its block and is evaluated once where it is written: a call read twice, read by a nested lambda or read after something else has run, an object that is created, a struct of the user's own or a boxed value, a mutable variable, and every let of a lambda that assigns or runs a statement. So does everything EvaluateQuotation and QuotationToExpression convert outside such a lambda. The lets the compiler adds for record fields written out of order are left out as a case of their own, in lambdas and, before the query translator looks at the record, in a select or join over IQueryable. The fields are then evaluated in declaration order, which is the one thing a lambda with side effects in its fields can notice. A join whose key is such a record on one side and comes ready-made from a function on the other keeps the key as written on both sides: the two no longer end up with different types. Fixes dotnet#11131 Fixes dotnet#15648 Fixes dotnet#11332 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A let is a record field that the compiler moved out of place only when its variable is named like the field whose place it takes. A let of the code with another name is evaluated first, like any other, even if a record follows it. A nullable is a plain value only when the type it holds is one, and the arguments of a method of a plain value must be plain too: a method can run code of the user's on an argument, once for every time it is made. A mutable let named like the copy the compiler makes of a struct gives way only for a plain value, so that a struct of the user's keeps its copy. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A let that adds, subtracts or multiplies constants and the data a lambda is given gives way to its value, like a let that only names a member: let next = p.Age + 1 no longer makes a block, and a lambda nested in the one that has it reads the value through the parameter. The operands are constants and reads, never arithmetic themselves, so a value that is built on another one does not grow with every let. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The lets that the compiler adds for record fields written out of order were
left out whatever their value was, so a field with a side effect was evaluated
after the fields that follow it: { Second = counter.Next(); First =
counter.Next() } gave First = 1, and so did a let written by hand and named
like the field it feeds. These lets are ordinary lets now. The construction
of the anonymous object that stands for the record stays behind them, so a
field that only reads still reaches the provider without a block.
A let of a call that makes a string was replaced by the call at every use, so
obj.ReferenceEquals(upper, upper) was false. It is replaced when it is used
once, or when every use only looks at what the string holds.
A division of reads by a constant other than 0 and -1 is a read as well, so the
lets of a key like { KeyAge = c.Age / 10; KeyCity = c.City } stay out of the
query.
The note on breaking changes lists what else changes when and how often a read
runs: in a nested lambda, after a call that changes its source, and on a path
that does not use it. It says what EvaluateQuotation does with a let inside a
lambda that goes to a provider. The release note names the join key that is
built on one side only, and the part of dotnet#3782 that this fixes.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A record selected with its fields written out of order is converted like one with its fields in order, and distinct compares the converted objects by their fields. For a record type with an Equals of its own that was already so when the fields were in order, and the test pins both orders to the same result. The breaking changes name it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
31c93a1 to
9e58c22
Compare
The note no longer credits dotnet#3782: the record with fields out of order is fixed by dotnet#20895. It names the distinct that returned a city once for each id, because the provider applied DISTINCT to both columns of the earlier projection. The note on breaking changes says that the SQL now names its columns, and that a groupBy on an item of a selected tuple needs dotnet#20899. A scenario takes one item of a tuple apart in the next for and applies distinct to it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Thanks @pcshrosbree. Changes since 31c93a1:
|

Fixes #11131
Fixes #15648
Fixes #11332
Stacked on #20894
A record or anonymous record with fields out of declaration order, a tupled lambda parameter,
x.Created.HasValueon a class property and aletthat names a member or is read once no longer reach LINQ providers asExpression.Block, which none of them translates.Details
lettakes the place of the variable where that computes the same: the value only reads (a constant, the lambda's parameter or a member of it, or what a member or method of a string, number or date makes of those), or the variable is read once, as the first thing the rest of the lambda evaluates.letkeeps its block and is evaluated once where it is written: a call read twice, by a nested lambda or after something else has run, an object that is created, a struct of your own or a boxed value, amutablevariable, and everyletof a lambda that assigns or runs a statement.p.Name.ToUpper(), keeps its block when it is used more than once and a use can tell one object from another, asobj.ReferenceEquals(upper, upper)does.joinwhose key one side builds as a tuple and the other side has ready-made no longer fails with "Type mismatch when building 'ty'".EvaluateQuotationandQuotationToExpressionoutside such a lambda still produce a block.🤖 Generated with Claude Code