NewFromFloat and NewFromFloat32 performance improvements - #431
Open
omegaatt36 wants to merge 1 commit into
Open
Conversation
NewFromFloat and NewFromFloat32 need the shortest decimal digit string
that round-trips back to the input float. strconv already computes
exactly that for FormatFloat with precision -1, but the library computes
it itself through decimal-go.go and rounding.go, a copy of the 2009
strconv/decimal.go: assign the mantissa into an 800-byte digit buffer,
shift it one binary bit at a time, then walk the digits against the two
neighbouring floats to find where they diverge.
Ask strconv for the value in 'e' format instead and read the digits back
out. The 'e' layout is [-]d[.ddd]e+dd, at most 24 bytes and trivial to
scan, unlike 'f' which can run past 750 bytes at the exponent extremes.
The result is identical, so no caller-visible behaviour changes.
Benchmarks (Ryzen 9 5900X, -benchtime 2s -count 5, median):
go1.22 go1.27
Float 366 -> 168 ns 332 -> 94 ns
Float32 271 -> 130 ns 241 -> 92 ns
Allocations are unchanged at 2 / 40 B, both from the big.NewInt that
Decimal.value requires. The gap widens on go1.27 because that release
replaced strconv's shortest-float conversion with the uscale algorithm.
Before go1.13, strconv.FormatFloat could return a shortest form that was
not the nearest one (golang/go#29491, the bug behind the two test table
entries added for it), so the old path is kept behind a !go1.13 build
tag. Nothing changes for go1.10 through go1.12; once the module's
minimum Go version passes 1.13, decimal-go.go, rounding.go and
newfromfloat_legacy.go can be deleted outright, removing 575 lines.
Verified equivalent against the previous implementation on go1.13 and
go1.27 over all 4,278,190,078 non-zero finite float32 bit patterns and
578,922,574 float64 values (random bit patterns, every power of two and
its neighbours, and a dense subnormal sweep), with no mismatch.
Tested on go1.10.8, go1.11.13, go1.12.17, go1.13.15, go1.22.12 and
go1.27.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
decimal-go.gois a copy ofstrconv/decimal.gofrom 2009.NewFromFloatandNewFromFloat32use this copy to find the shortest decimal string for a float. The Go standard library no longer uses this code. Go 1.27 replaced it with a new algorithm. The vendored copy does not get these improvements.This PR removes the vendored path from
NewFromFloatandNewFromFloat32. The functions now callstrconv.FormatFloat(v, 'e', -1, bitSize). This call returns the shortest decimal string that round-trips to the original float. The function then parses the digits and the exponent from that string. The exponent is parsed with a small loop.strconv.Atoiwould need astring(byteslice)conversion and one more allocation.The output does not change. The shortest round-trip contract of
NewFromFloatstays the same. See #411 and #418 for the current discussion about this contract. This PR does not change the contract. It only makes the same output faster.Benchmarks
Ryzen 9 5900X, Linux,
-benchtime 2s -count 5, median. The machine was idle.NewFromFloatNewFromFloat32The change in Go 1.27 is larger because Go 1.27 replaced the float conversion code in
strconvwith the new uscale algorithm. The vendored copy in this repo could not benefit from that change. This PR connectsNewFromFloattostrconvagain.Both versions allocate the same: 2 allocs / 40 B. Both allocations come from the
*big.IntinDecimal.value. This is a property of theDecimaltype, not of this change.Correctness
I compared the old implementation with the new one on the same inputs. The functions must return the same
(value, exp)pair.I ran the comparison on Go 1.13 (the first version that uses the new path) and on Go 1.27. Both runs show 0 mismatches.
The test suite passes on six toolchains:
The PR adds two tests in
newfromfloat_go113_test.go:TestNewFromFloatShortestRoundTrip. The result must equalNewFromString(strconv.FormatFloat(f, 'f', -1, bitSize)), andFloat64()must return the original float. The test covers powers of two with neighbors, three-digit exponents, and range extremes.TestNewFromFloatRandomRoundTrip. 200,000 random float64 bit patterns and 200,000 random float32 bit patterns.These tests use
strconvas the reference. On Go 1.12 and older,strconvitself gives wrong answers in rare cases. See the next section. This is why the new tests are gated with//go:build go1.13.Why build tags, not a go.mod bump: golang/go#29491
Before Go 1.13, the shortest-decimal code in
strconvsometimes picked the candidate that is farther from the exact value. Two test cases indecimal_test.goexist for this reason. They are marked with a comment about issue golang/go#29491. The vendored slow path gives the correct answer there.strconv.FormatFloatwith-1gives the wrong answer on Go 1.10, 1.11, and 1.12. Go 1.13 fixed this. I verified the boundary on all three old toolchains.Example from the probe:
For this reason the old implementation moves to
newfromfloat_legacy.gobehind//go:build !go1.13.decimal-go.goandrounding.goget a!go1.13build tag too. Their content does not change. Files with ago1.13tag do not need their content.The go.mod version stays at 1.10. I read the discussions in #305, #353, and #361. The project keeps the oldest Go version that it can support. The build tags keep that property. The CI matrix already runs 1.10.x, and the tests pass there through the legacy path.
When the go.mod minimum reaches 1.13 in the future, the files
decimal-go.go,rounding.go, andnewfromfloat_legacy.gocan be deleted. That removes 575 lines.File changes
Each build tag carries both
//go:buildand the old// +buildform.decimal_go124_test.goalready uses this pattern.newfromfloat_legacy.gokeeps themath.Float32bits(value) ^ 0x80808080workaround for golang/go#26285. The behavior on Go 1.12 and older stays exactly the same as on master.Background
I wrote a detailed analysis of the Go 1.27
strconvrewrite and its effect on libraries that vendor the old code (in Chinese): https://www.omegaatt.com/blogs/develop/2026/golang_unrounded_scaling/Related issues: #161 (rounding bug in the vendored
roundShortest) is another cost of keeping the vendored copy.