Skip to content

NewFromFloat and NewFromFloat32 performance improvements - #431

Open
omegaatt36 wants to merge 1 commit into
shopspring:masterfrom
omegaatt36:perf/newfromfloat-strconv
Open

NewFromFloat and NewFromFloat32 performance improvements#431
omegaatt36 wants to merge 1 commit into
shopspring:masterfrom
omegaatt36:perf/newfromfloat-strconv

Conversation

@omegaatt36

Copy link
Copy Markdown

Summary

decimal-go.go is a copy of strconv/decimal.go from 2009. NewFromFloat and NewFromFloat32 use this copy to find the shortest decimal string for a float. The Go standard library no longer uses this code. Go 1.27 replaced it with a new algorithm. The vendored copy does not get these improvements.

This PR removes the vendored path from NewFromFloat and NewFromFloat32. The functions now call strconv.FormatFloat(v, 'e', -1, bitSize). This call returns the shortest decimal string that round-trips to the original float. The function then parses the digits and the exponent from that string. The exponent is parsed with a small loop. strconv.Atoi would need a string(byteslice) conversion and one more allocation.

The output does not change. The shortest round-trip contract of NewFromFloat stays the same. See #411 and #418 for the current discussion about this contract. This PR does not change the contract. It only makes the same output faster.

Benchmarks

Ryzen 9 5900X, Linux, -benchtime 2s -count 5, median. The machine was idle.

Benchmark Go 1.22 before Go 1.22 after Go 1.27 before Go 1.27 after
NewFromFloat 366.4 ns 167.7 ns 332.2 ns 93.7 ns
NewFromFloat32 271.8 ns 130.0 ns 241.7 ns 91.9 ns

The change in Go 1.27 is larger because Go 1.27 replaced the float conversion code in strconv with the new uscale algorithm. The vendored copy in this repo could not benefit from that change. This PR connects NewFromFloat to strconv again.

Both versions allocate the same: 2 allocs / 40 B. Both allocations come from the *big.Int in Decimal.value. This is a property of the Decimal type, not of this change.

Correctness

I compared the old implementation with the new one on the same inputs. The functions must return the same (value, exp) pair.

  • float32: all 4,278,190,078 non-zero finite bit patterns. Exhaustive, not a sample. Mismatches: 0.
  • float64: 578,922,574 random and structured bit patterns. This covers subnormals, extreme exponents, powers of two with their neighbors (exponents -1074 to 1023), powers of ten (exponents -340 to 308), and dense scans of low subnormals. Mismatches: 0.

I ran the comparison on Go 1.13 (the first version that uses the new path) and on Go 1.27. Both runs show 0 mismatches.

The test suite passes on six toolchains:

go1.10.8   ok   (legacy path)
go1.11.13  ok   (legacy path)
go1.12.17  ok   (legacy path)
go1.13.15  ok   (new path)
go1.22.12  ok   (new path)
go1.27.0   ok   (new path)

The PR adds two tests in newfromfloat_go113_test.go:

  • TestNewFromFloatShortestRoundTrip. The result must equal NewFromString(strconv.FormatFloat(f, 'f', -1, bitSize)), and Float64() must return the original float. The test covers powers of two with neighbors, three-digit exponents, and range extremes.
  • TestNewFromFloatRandomRoundTrip. 200,000 random float64 bit patterns and 200,000 random float32 bit patterns.

These tests use strconv as the reference. On Go 1.12 and older, strconv itself gives wrong answers in rare cases. See the next section. This is why the new tests are gated with //go:build go1.13.

Why build tags, not a go.mod bump: golang/go#29491

Before Go 1.13, the shortest-decimal code in strconv sometimes picked the candidate that is farther from the exact value. Two test cases in decimal_test.go exist for this reason. They are marked with a comment about issue golang/go#29491. The vendored slow path gives the correct answer there. strconv.FormatFloat with -1 gives the wrong answer on Go 1.10, 1.11, and 1.12. Go 1.13 fixed this. I verified the boundary on all three old toolchains.

Example from the probe:

498484681984085570  (bits 0x439babe4b56e8a39)
  exact value:        498484681984085568
  Go 1.10-1.12:      4.9848468198408556e+17   distance 8    (wrong)
  Go 1.13 and later: 4.9848468198408557e+17   distance 2    (correct)

For this reason the old implementation moves to newfromfloat_legacy.go behind //go:build !go1.13. decimal-go.go and rounding.go get a !go1.13 build tag too. Their content does not change. Files with a go1.13 tag do not need their content.

The go.mod version stays at 1.10. I read the discussions in #305, #353, and #361. The project keeps the oldest Go version that it can support. The build tags keep that property. The CI matrix already runs 1.10.x, and the tests pass there through the legacy path.

When the go.mod minimum reaches 1.13 in the future, the files decimal-go.go, rounding.go, and newfromfloat_legacy.go can be deleted. That removes 575 lines.

File changes

decimal.go                  -50   two call sites remain: newFromFloat(value, 64) / (float64(value), 32)
newfromfloat_go113.go       +70   //go:build go1.13    new strconv path
newfromfloat_legacy.go      +76   //go:build !go1.13   old newFromFloat, moved
decimal-go.go               +7    //go:build !go1.13   build tag only, no content change
rounding.go                 +7    //go:build !go1.13   build tag only, no content change
newfromfloat_go113_test.go  +114  //go:build go1.13    new tests

Each build tag carries both //go:build and the old // +build form. decimal_go124_test.go already uses this pattern.

newfromfloat_legacy.go keeps the math.Float32bits(value) ^ 0x80808080 workaround for golang/go#26285. The behavior on Go 1.12 and older stays exactly the same as on master.

Background

I wrote a detailed analysis of the Go 1.27 strconv rewrite and its effect on libraries that vendor the old code (in Chinese): https://www.omegaatt.com/blogs/develop/2026/golang_unrounded_scaling/

Related issues: #161 (rounding bug in the vendored roundShortest) is another cost of keeping the vendored copy.

NewFromFloat and NewFromFloat32 need the shortest decimal digit string
that round-trips back to the input float. strconv already computes
exactly that for FormatFloat with precision -1, but the library computes
it itself through decimal-go.go and rounding.go, a copy of the 2009
strconv/decimal.go: assign the mantissa into an 800-byte digit buffer,
shift it one binary bit at a time, then walk the digits against the two
neighbouring floats to find where they diverge.

Ask strconv for the value in 'e' format instead and read the digits back
out. The 'e' layout is [-]d[.ddd]e+dd, at most 24 bytes and trivial to
scan, unlike 'f' which can run past 750 bytes at the exponent extremes.
The result is identical, so no caller-visible behaviour changes.

Benchmarks (Ryzen 9 5900X, -benchtime 2s -count 5, median):

               go1.22           go1.27
  Float    366 -> 168 ns    332 ->  94 ns
  Float32  271 -> 130 ns    241 ->  92 ns

Allocations are unchanged at 2 / 40 B, both from the big.NewInt that
Decimal.value requires. The gap widens on go1.27 because that release
replaced strconv's shortest-float conversion with the uscale algorithm.

Before go1.13, strconv.FormatFloat could return a shortest form that was
not the nearest one (golang/go#29491, the bug behind the two test table
entries added for it), so the old path is kept behind a !go1.13 build
tag. Nothing changes for go1.10 through go1.12; once the module's
minimum Go version passes 1.13, decimal-go.go, rounding.go and
newfromfloat_legacy.go can be deleted outright, removing 575 lines.

Verified equivalent against the previous implementation on go1.13 and
go1.27 over all 4,278,190,078 non-zero finite float32 bit patterns and
578,922,574 float64 values (random bit patterns, every power of two and
its neighbours, and a dense subnormal sweep), with no mismatch.

Tested on go1.10.8, go1.11.13, go1.12.17, go1.13.15, go1.22.12 and
go1.27.0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant