Skip to content

fix(storage): optimize CRC32C#update using Slicing-by-8 algorithm - #9711

Open
ddelgrosso1 wants to merge 5 commits into
googleapis:mainfrom
ddelgrosso1:perf-crc32c
Open

ddelgrosso1 wants to merge 5 commits into
googleapis:mainfrom
ddelgrosso1:perf-crc32c

Conversation

@ddelgrosso1

Copy link
Copy Markdown
Contributor

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

@ddelgrosso1
ddelgrosso1 requested review from a team as code owners October 23, 2026 14:01
@product-auto-label product-auto-label Bot added the api: storage Issues related to the Cloud Storage API. label Oct 23, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes the CRC32C calculation by implementing a slicing-by-8 lookup table algorithm in the update method, processing 8 bytes at a time, and adds extensive unit tests to verify correctness across various scenarios (RFC test vectors, unaligned views, remainder lengths, etc.). Feedback suggests optimizing the loop condition in the hot path by pre-calculating the loop limit to avoid addition in each iteration.

Comment thread handwritten/storage/src/crc32c.ts Resolved
@ddelgrosso1

ddelgrosso1 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author

I ran a local benchmark (not against live GCS) stubbing the HTTP layer (startResumableUpload_ and requestStream) with in-memory streams and piping 64 KiB chunks through createWriteStream and createReadStream with CRC32C validation enabled.

In-memory stream + CRC32C processing speed went from ~216 MiB/s to ~1,070 MiB/s (~5x faster), cutting time per GiB from ~4.7s down to ~0.95s.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: storage Issues related to the Cloud Storage API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant