-
Notifications
You must be signed in to change notification settings - Fork 3.6k
Add streaming (constant-memory) Xlsx writer #4966
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
kemo
wants to merge
37
commits into
PHPOffice:master
Choose a base branch
from
kemo:streaming-writer
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,762
−0
Open
Changes from 33 commits
Commits
Show all changes
37 commits
Select commit
Hold shift + click to select a range
36a446f
Add StreamedCell value object for streaming writer
kemo 0412d0a
Add StreamingWriter skeleton with zip assembly and shell workbook parts
kemo 3c87d76
Add appendRow with scalar types and string handling to streaming writer
kemo 36f6998
Use inline strings for streaming writer string cells
kemo d7e248f
Add formula support with full calc on load to streaming writer
kemo 5095ba7
Add DateTime support with default date format to streaming writer
kemo 2121367
Add style round-trip tests for streaming writer
kemo 2065c8b
Add column widths, freeze pane and autofilter to streaming writer
kemo a7be151
Validate freeze pane state before the A1 no-op
kemo 230b775
Add lifecycle guard tests for streaming writer
kemo 3efb9b0
Use unqualified Exception import in streaming writer
kemo 2524a85
Add flat-memory guard test for streaming writer
kemo dde8440
Assert streaming writer memory is flat past zip block saturation
kemo 9b2c4a3
Add streaming writer documentation
kemo 607f0f7
Fix new phpstan errors in streaming writer
kemo 518c2ff
Fix always-on forceFullCalc and add streaming writer failure cleanup
kemo 64f2682
Guard streaming sheet against corruption from a failed appendRow()
kemo 0482153
Add tests for the streaming writer fix wave
kemo eda22b5
Correct streaming writer documentation
kemo 4c2933c
Add streaming writer benchmark scripts
kemo 1aff9f7
Link changelog entry to PR 4966
kemo 84c0414
Reject non-finite numbers, invalid UTF-8 and over-limit strings
kemo 0adaab1
Cover StreamedCell null, forced-string UTF-8 and freeze pane no-op paths
kemo 2a9b6a8
Surface close() failures as writer exceptions and guard the output ha…
kemo 56816c2
Use stringValue helper instead of casting mixed in streaming tests
kemo 275731c
Validate sheet names in startSheet before mutating the shell workbook
kemo 4c2d346
Match standard writer numeric serialization in streaming sheets
kemo a7e53de
Wrap unexpected appendRow failures in writer exceptions
kemo 851a3f0
Validate and normalize the freeze pane cell eagerly
kemo da10704
Document streaming writer name rules, binder-free values, and cleanup
kemo aa7224c
Merge branch 'master' into streaming-writer
kemo 29a5315
Guard memory_reset_peak_usage with function_exists instead of a phpst…
kemo fcf2b4b
Merge remote streaming-writer (upstream master sync) into local branch
kemo 2badc79
Use the unambiguous datetime format for default date styles
kemo 822057f
Collapse duplicate constructor throw into the false-handle check
kemo 729d1b0
Explain the forceFullCalc argument at the workbook write site
kemo 7f67587
Explain why freezePane at A1 writes no pane
kemo File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,233 @@ | ||
| # Streaming Xlsx writer | ||
|
|
||
| `PhpOffice\PhpSpreadsheet\Writer\Xlsx\Streaming\StreamingWriter` writes Xlsx | ||
| files directly to a file handle, one row at a time. It does not build a | ||
| `Spreadsheet` object in memory. Use it for large exports where the standard | ||
| writer would use too much memory. | ||
|
|
||
| ## When to use it | ||
|
|
||
| Use the streaming writer for large, append-only exports, for example a | ||
| report with hundreds of thousands of rows. Peak memory does not depend on | ||
| the number of rows written. | ||
|
|
||
| Measurements on this feature: writing 100,000 rows and writing 200,000 rows | ||
| both peak at about 34MB of total memory. Most of that 34MB is a fixed 16MB | ||
| read block used internally by the Zip stream, not row data. The standard | ||
| `Xlsx` writer holds every cell in memory, at roughly 1KB or more per cell, | ||
| so its memory use grows with the row count. | ||
|
|
||
| Do not use the streaming writer when you need to read the file back, edit | ||
| existing cells, or use features it does not support (see below). Use the | ||
| standard `Xlsx` writer, with [cell caching](memory_saving.md) if needed, | ||
| for those cases. | ||
|
|
||
| ## Basic usage | ||
|
|
||
| ```php | ||
| use PhpOffice\PhpSpreadsheet\Writer\Xlsx\Streaming\StreamingWriter; | ||
|
|
||
| $writer = new StreamingWriter('large-export.xlsx'); | ||
| $sheet = $writer->startSheet('Data'); | ||
| $sheet->appendRow(['Name', 'Amount']); | ||
| $sheet->appendRow(['Widget', 12.5]); | ||
| $writer->close(); | ||
| ``` | ||
|
|
||
| Rows are plain arrays of values, written left to right starting at column A. | ||
| A `null` value leaves that cell empty. Once `close()` has run, the writer | ||
| and every sheet it produced are no longer usable. | ||
|
|
||
| ## Full worked example | ||
|
|
||
| This example shows styles, dates, formulas, column widths, a freeze pane, | ||
| an autofilter, and a second sheet: | ||
|
|
||
| ```php | ||
| use PhpOffice\PhpSpreadsheet\Writer\Xlsx\Streaming\StreamedCell; | ||
| use PhpOffice\PhpSpreadsheet\Writer\Xlsx\Streaming\StreamingWriter; | ||
|
|
||
| $writer = new StreamingWriter('large-export.xlsx'); | ||
|
|
||
| // Styles are registered once on the writer and referenced by id. | ||
| $headerStyle = $writer->registerStyle(['font' => ['bold' => true]]); | ||
| $moneyStyle = $writer->registerStyle(['numberFormat' => ['formatCode' => '#,##0.00']]); | ||
|
|
||
| $sheet = $writer->startSheet('Orders'); | ||
|
|
||
| // Column widths and the freeze pane must be set before the first | ||
| // appendRow() call. | ||
| $sheet->setColumnWidths([1 => 30.0, 2 => 15.0, 3 => 12.0]); | ||
| $sheet->freezePane('A2'); | ||
|
|
||
| // The autofilter can be requested at any time before the sheet is | ||
| // finished; the actual range is only known once the sheet finishes. | ||
| $sheet->setAutoFilterToWrittenRange(); | ||
|
|
||
| // A style id applies to every cell in the row unless overridden per cell. | ||
| $sheet->appendRow(['Customer', 'Order Date', 'Total'], $headerStyle); | ||
|
|
||
| $sheet->appendRow([ | ||
| 'Acme Corp', | ||
| new \DateTimeImmutable('2026-01-15'), | ||
| new StreamedCell(1234.5, $moneyStyle), | ||
| ]); | ||
|
|
||
| // A leading "=" writes a formula; it is stored without a cached value. | ||
| // A string that is only "=" (length 1) is not long enough to be treated | ||
| // as a formula and is written as a plain string instead. | ||
| $sheet->appendRow(['Total', null, '=SUM(C2:C2)']); | ||
|
|
||
| // Start a second sheet; the first sheet is finished and can no longer | ||
| // be appended to once startSheet() runs again. | ||
| $sheet2 = $writer->startSheet('Notes'); | ||
| $sheet2->appendRow(['Generated by the streaming writer.']); | ||
|
|
||
| $writer->close(); | ||
| ``` | ||
|
|
||
| Because the workbook contains a formula, `close()` marks the workbook for | ||
| full calculation on load, so Excel (or another spreadsheet application) | ||
| computes `=SUM(C2:C2)` when it opens the file. The written cell itself has | ||
| no cached value. If no sheet in the workbook contains a formula, `close()` | ||
| does not force a recalculation on load. | ||
|
|
||
| Any `DateTimeInterface` value that has no explicit style gets a default | ||
| date number format automatically. | ||
|
|
||
| `freezePane('A1')` is a no-op; freezing at the top-left cell freezes | ||
|
kemo marked this conversation as resolved.
Outdated
|
||
| nothing. | ||
|
|
||
| Each finished sheet keeps its buffered XML in memory (a `php://temp` | ||
| stream) until `close()` runs, up to about 2MB per sheet before it spills | ||
| to a real temporary file on disk. Memory use at `close()` therefore scales | ||
| with the number of sheets, not with the number of rows in any one sheet. | ||
|
|
||
| ## API | ||
|
|
||
| ### `StreamingWriter` | ||
|
|
||
| - `__construct(string $filename)` — opens `$filename` for writing. | ||
| - `startSheet(string $name): StreamingSheet` — finishes the current sheet, | ||
| if any, and starts a new one. The name must not be empty, must be at most | ||
| 31 characters, and must not contain `* : / \ ? [ ]`; an invalid name | ||
| throws a `Writer\Exception` and leaves the writer usable. A name that is | ||
| already in use is silently given a numeric suffix (a second `Data` sheet | ||
| becomes `Data 1`), the same behavior as `Worksheet::setTitle()`. | ||
| - `registerStyle(array $styleArray): int` — registers a style, in the same | ||
| array format used by `Style::applyFromArray()`, and returns its style id. | ||
| - `close(): void` — finishes the last sheet and writes the Xlsx file. The | ||
| writer must have at least one sheet. | ||
|
|
||
| ### `StreamingSheet` (returned by `startSheet()`) | ||
|
|
||
| - `appendRow(array $cells, ?int $styleId = null): void` — writes one row. | ||
| `$styleId`, if given, applies to every cell in the row that does not | ||
| carry its own style through `StreamedCell`. | ||
| - `setColumnWidths(array $widths): void` — `$widths` is a 1-based column | ||
| number mapped to a width, for example `[1 => 30.0, 3 => 12.0]`. Must be | ||
| called before the first `appendRow()`. | ||
| - `freezePane(string $cell): void` — freezes rows and columns above and to | ||
| the left of `$cell`, for example `'A2'` to freeze the header row. Must be | ||
| called before the first `appendRow()`. The cell reference is validated | ||
| immediately; an invalid reference throws a `Writer\Exception` and leaves | ||
| the sheet usable. | ||
| - `setAutoFilterToWrittenRange(): void` — adds an autofilter over the full | ||
| range written to the sheet. Can be called at any time before the sheet is | ||
| finished; the range is only known once the sheet is finished. | ||
|
|
||
| ### `StreamedCell` | ||
|
|
||
| Wrap a single cell value in `new StreamedCell($value, $styleId, $dataType)` | ||
| to give that cell its own style, overriding the row style, or to force its | ||
| data type. `$styleId` and `$dataType` both default to `null`. The only | ||
| supported `$dataType` values are `DataType::TYPE_STRING` and | ||
| `DataType::TYPE_STRING2`, which force the value to be written as a string | ||
| even if it looks like a formula (starts with `=`). Any other `$dataType` | ||
| value throws a `Writer\Exception`. Forcing a non-scalar value, such as a | ||
| `DateTimeInterface` date, to a string also throws; only scalar values can | ||
| be forced to string. | ||
|
|
||
| ### Supported cell values | ||
|
|
||
| - Strings | ||
| - Integers and floats | ||
| - Booleans | ||
| - `DateTimeInterface` instances | ||
| - Formula strings (any string starting with `=`) | ||
| - `null` (leaves the cell empty) | ||
|
|
||
| Values are written by their PHP type; there is no value binder. A numeric | ||
| string such as `'123'` is written as text, not as a number. Pass an `int` | ||
| or a `float` to write a number. | ||
|
|
||
| Any other value type throws a `Writer\Exception`. Invalid values of a | ||
| supported type also throw instead of producing a broken file: | ||
|
|
||
| - `NAN` and `INF` floats (Excel cannot store non-finite numbers) | ||
| - Strings that are not valid UTF-8 (they would corrupt the sheet XML) | ||
| - Strings longer than 32,767 characters (the Excel cell limit) | ||
|
|
||
| ## Strings are written as inline strings | ||
|
|
||
| The streaming writer has no shared string table; every string cell is | ||
| written as an inline string. Excel and other spreadsheet applications read | ||
| inline strings as plain strings, with no special handling needed. | ||
|
|
||
| PhpSpreadsheet's own `Xlsx` reader is the exception: it reads inline string | ||
| cells back as `RichText` objects, not plain strings. If you read a | ||
| streaming-writer file back with PhpSpreadsheet, cast the cell value to | ||
| `(string)` before comparing it to the original string: | ||
|
|
||
| ```php | ||
| $value = (string) $worksheet->getCell('A1')->getValue(); | ||
| ``` | ||
|
|
||
| ## Supported features | ||
|
|
||
| - Values, formulas, and dates | ||
| - Per-row styles (the `$styleId` argument of `appendRow()`) | ||
| - Per-cell styles and forced data types (`StreamedCell`) | ||
| - Column widths (`setColumnWidths()`) | ||
| - A single freeze pane (`freezePane()`) | ||
| - An autofilter over the written range (`setAutoFilterToWrittenRange()`) | ||
| - Multiple sheets, written one after another (`startSheet()`) | ||
|
|
||
| ## Not supported | ||
|
|
||
| The streaming writer is append-only and forward-only. It does not support: | ||
|
|
||
| - Random access: you cannot go back and change a cell or row already | ||
| written. | ||
| - Reading the file back through this writer; use the `Xlsx` reader instead, | ||
| and remember the inline-string caveat above. | ||
| - Merged cells | ||
| - Charts | ||
| - Drawings and images | ||
| - Comments | ||
| - Conditional formatting | ||
| - Hyperlinks | ||
| - Rich text cell values | ||
|
|
||
| ## Lifecycle rules | ||
|
|
||
| - `setColumnWidths()` and `freezePane()` must be called before the first | ||
| `appendRow()` on that sheet. Calling either after the first row throws a | ||
| `Writer\Exception`. | ||
| - Sheets are sequential. Calling `startSheet()` finishes the current sheet; | ||
| the `StreamingSheet` object returned by the previous `startSheet()` call | ||
| becomes unusable, and any further method call on it throws a | ||
| `Writer\Exception`. | ||
| - Nothing on the writer or on any sheet is usable after `close()` has run. | ||
| - Arguments are checked before any XML is written where possible: an | ||
| invalid sheet name, an unregistered row-level `$styleId`, an invalid | ||
| freeze pane cell, and invalid column widths all throw without harming | ||
| the writer, so those exceptions are safe to catch and correct. | ||
| - A failed `appendRow()` call that throws while a row is being built, for | ||
| example because of an unsupported value or an unregistered | ||
| `StreamedCell` style id, invalidates the sheet. Every later call on that | ||
| sheet, including `close()`, throws. Discard the writer and start again. | ||
| - If the writer is destroyed without a `close()` call, for example when an | ||
| exception ends the request, the destructor deletes the partial file. No | ||
| file exists on disk until `close()` returns; a forgotten `close()` means | ||
| no output and no error. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| <?php | ||
|
|
||
| declare(strict_types=1); | ||
|
|
||
| namespace PhpOffice\PhpSpreadsheet\Writer\Xlsx\Streaming; | ||
|
|
||
| /** | ||
| * Immutable per-cell wrapper for StreamingSheet::appendRow(), | ||
| * used to attach a style id or force a data type. | ||
| */ | ||
| final class StreamedCell | ||
| { | ||
| public function __construct( | ||
| public readonly mixed $value, | ||
| public readonly ?int $styleId = null, | ||
| public readonly ?string $dataType = null, | ||
| ) { | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do not see where
closemarks the workbook for full calculation. Please enlighten me.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment* updated 729d1b0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still don't see where that attribute is set in your code. Please explain.
More to the point, 'forceFullCalc` is a very dangerous option. Please see PR #4271 for details on its unexpected side effects, and why we would not want to make it an "automatic" choice.