tilo · tilo · Aug 10, 2026 · Jul 22, 2026 · Aug 9, 2026 · Aug 9, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,68 @@
 > [!TIP]
 > **Upgrading?** The [SmarterCSV Upgrade Wizard](https://tilo.github.io/smarter_csv/upgrade_wizard.html) walks you through what (if anything) you need to change for your specific version. Most steps do not require any changes.
 
+## 1.19.0 (2026-08-10)
+
+RSpec tests: **2,595 → 3,164** (+569 tests)
+
+
+### Reverted Behavior Changes
+
+  - **Exponent forms are no longer auto-converted to numbers ([#345](https://github.com/tilo/smarter_csv/issues/345)).**
+
+  Version 1.18.0 started converting scientific notation (`"1e3"`, `"12E5"`, `"1.5e3"`) to Floats. In real-world CSV data, digits-E-digits values are far more often identifiers (short codes, hex IDs) than scientific notation, and the auto-conversion corrupted them irreversibly — an ID like `"0047583311587E590003"` came back as `Infinity`.
+
+  As of 1.19.0, exponent forms always stay Strings — as they did in every version before 1.18.0. If a column really does contain scientific notation, convert it per-column with `value_converters`:
+
+  ```ruby
+  SmarterCSV.process(file, value_converters: { measurement: ->(v) { v.to_f } })
+  ```
+
+  Thanks to [Denis Sadomowski](https://github.com/sonicdes) for the report.
+
+### Behavior Changes
+
+  - **`field_size_limit` values below `4096` now raise a `ValidationError`** — the option is overrun protection (a hard upper bound against runaway fields), not per-field validation.
+
+  - **The Hash form of `convert_values_to_numeric` is now validated and normalized.** It requires exactly one of `only:`/`except:` with field name(s) (String/Symbol or an Array of them); an empty hash, unknown keys, both keys together, empty lists, or `nil`/boolean values raise a `ValidationError` (the two parser paths previously disagreed on these shapes). The listed names are normalized to the row-key type, so `only:`/`except:` now also works with `strings_as_keys` / `keep_original_headers`.
+
+  - **All empty field values are ONE shared, frozen, UTF-8 empty-string object — on both paths** (no String allocation per empty field). Mutating an empty value now raises `FrozenError` instead of silently changing every other empty value in the result. Relevant with `remove_empty_values: false`; the default `true` removes empties anyway.
+
+### Bug Fixes
+
+  - **Writer**: fields needing quoting are wrapped in the configured `quote_char`, not a hard-coded `"` — output with a custom `quote_char` round-trips again.
+  - **`Reader#each` without a block** no longer clears the configured `chunk_size` for a later `each_chunk` on the same Reader.
+  - **Duplicate-header disambiguation** no longer steals a real column's name — `name,name,name2` no longer raises `DuplicateHeaders` (the second `name` becomes `name3`).
+  - **The caller's `user_provided_headers` array** (and a reused options hash) is no longer mutated when rows have more columns than headers.
+  - **`headers: { only: }` / `{ except: }`** now works with `strings_as_keys` / `keep_original_headers` — selectors are normalized to the row-key type (with `only:`, nothing matched and every row came back empty: silent total data loss).
+  - **A quoted header containing an embedded newline** is stitched across physical lines like data rows (`"first\nname"` → `:first_name`); previously the first fragment was silently lost. An unclosed header quote at end-of-file raises `MalformedCSV`.
+  - **`quote_char: :auto`** raises a `ValidationError` instead of crashing with `NoMethodError` — quote_char has no auto-detection.
+
+### Bug Fixes — C/Ruby parity
+
+  The C-accelerated and pure-Ruby parsers now behave identically in all of the following cases (same input, same output — verified by differential fuzzing and by running every parsing spec on both paths):
+
+  - a partial multi-char separator at end-of-line is field content, not a separator — with `col_sep: '||'` the C path silently dropped the lone `|` from `"y|"` (also fixes an out-of-bounds read near end-of-line)
+  - a trailing `\r` before an LF row separator is part of the line terminator — a CRLF line with a quoted last field raised `MalformedCSV` on the C path, and `strip_whitespace: false` kept `"x\r"` / `"1\r"` as values
+  - with `strip_whitespace: true`, values are stripped of Ruby's full `String#strip` character set on the C path too (a stray `\r` from mixed LF/CRLF files survived before)
+  - an empty line yields `nil` for ALL columns with `remove_empty_values: false` (the C path gave the first column `""`)
+  - a row consisting only of NUL bytes counts as blank on the C path too (`String#strip` semantics; the NUL byte itself remains data with `remove_empty_hashes: false`)
+  - a `nil` entry in `user_provided_headers` drops that column on the C path too
+  - an empty-string header key (`strings_as_keys` + `duplicate_header_suffix: nil`) is dropped on the C path too
+  - non-ASCII `missing_header_prefix` (e.g. `"spalte_ä_"`) no longer raises `EncodingError` on the C path — extra-column keys are interned as UTF-8 symbols
+  - `col_sep` / `row_sep` / `missing_header_prefix` values longer than the C parser's internal buffers fall back to the pure-Ruby parser instead of being silently truncated
+  - `nil_values_matching` matches the RAW string value on the C path (a pattern like `/\A007\z/` only ever saw the converted `7`) — and no longer switches off numeric conversion and zero-removal for the non-matching values
+  - `field_size_limit` is checked against the raw field size BEFORE numeric conversion, so an oversized digit-only field raises on the C path too instead of being converted to a huge Integer (the exact overrun the option exists to prevent)
+  - `headers: { only: }` short-cuts on the pure-Ruby path too — parsing stops right after the last wanted column, matching the C path (no `:column_N` discovery behind it, and faster)
+  - a one-character multi-byte `col_sep` (e.g. `'é'`) no longer crashes the pure-Ruby parser
+  - a multi-byte character directly before a literal quote (`é"x`) no longer crashes the pure-Ruby parser (`IndexError` from a mid-character byte offset)
+  - invalid bytes in the input (typically Latin-1 data mislabeled as UTF-8) no longer crash the pure-Ruby parser — fields keep their raw bytes and encoding tag exactly (never transcoded, so the data stays recoverable via `force_encoding`); cleanup remains opt-in via `force_utf8` / `invalid_byte_sequence`
+  - the multiline stitch gate models the parser's rules exactly (doubled-quote precedence, backslash escapes, end-of-line chomp) — no more fabricated `MalformedCSV` on the pure-Ruby path for rows the parser can close
+
+### Tests
+
+  - **Every parsing spec now runs on BOTH the C-accelerated and the pure-Ruby path** via `[true, false]` acceleration loops (~420 additional examples), a seeded differential parity-fuzz spec (2,000 randomized inputs per run, including combined option sets) guards C/Ruby parity permanently, and line coverage is at **100%**.
+
 ## 1.18.1 (2026-06-30)
 
 ### Bug Fixes

diff --git a/CONTRIBUTORS.md b/CONTRIBUTORS.md
@@ -1,4 +1,4 @@
-# A Big Thank You to all 65 Contributors!!
+# A Big Thank You to all 66 Contributors!!
 
 
 A Big Thank you to everyone who filed issues, sent comments, and who contributed with pull requests:
@@ -68,3 +68,4 @@ A Big Thank you to everyone who filed issues, sent comments, and who contributed
  * [Jonas Staškevičius](https://github.com/pirminis)
  * [conorg](https://github.com/conorg)
  * [Alex Shenia](https://github.com/alexshenia)
+ * [Denis Sadomowski](https://github.com/sonicdes)
diff --git a/README.md b/README.md
@@ -364,7 +364,7 @@ For reporting issues, please:
   * open a pull-request adding a test that demonstrates the issue
   * mention your version of SmarterCSV, Ruby, Rails
 
-# [A Special Thanks to all 65 Contributors!](CONTRIBUTORS.md) 🎉🎉🎉
+# [A Special Thanks to all Contributors!](CONTRIBUTORS.md) 🎉🎉🎉
 
 
 ## Contributing

diff --git a/docs/bad_row_quarantine.md b/docs/bad_row_quarantine.md
@@ -266,7 +266,9 @@ until it either finds the closing quote or reaches end-of-file, potentially cons
 of megabytes.
 
 `field_size_limit` sets a hard cap (in bytes) on the size of any individual extracted field.
-The default is `nil` (no limit). When a field exceeds the limit a
+The default is `nil` (no limit); the minimum allowed value is `4096` — this option is overrun
+protection against runaway or crafted fields, not a per-field validation tool, so small values
+are rejected with a `ValidationError`. When a field exceeds the limit a
 `SmarterCSV::FieldSizeLimitExceeded` exception is raised — and because it inherits from
 `SmarterCSV::Error`, the `on_bad_row` option handles it exactly like any other parse error.
 

diff --git a/docs/basic_read_api.md b/docs/basic_read_api.md
@@ -223,7 +223,7 @@ comment_regexp → strip_chars_from_headers → split on col_sep → strip quote
     → disambiguate_headers → symbolize → key_mapping
 ```
 
-`user_provided_headers` bypasses the file header and all transformation steps — your array is used as-is.
+`user_provided_headers` bypasses the file header and all transformation steps — your array is used as-is; a `nil` entry drops that column.
 
 See [Header Transformations](./header_transformations.md) for the full step-by-step table and options.
 

diff --git a/docs/data_transformations.md b/docs/data_transformations.md
@@ -114,6 +114,8 @@ data = SmarterCSV.process(file, remove_empty_values: false)
 # => [{name: "Alice", score: 42, notes: nil}, {name: nil, score: nil, notes: "great player"}]
 ```
 
+With `remove_empty_values: false`, all kept empty-string values are ONE shared frozen String (an allocation optimization) — `dup` before mutating one in place *(1.19.0+)*.
+
 ---
 
 ## `remove_zero_values`
@@ -156,15 +158,17 @@ data = SmarterCSV.process(file,
   convert_values_to_numeric: { only: [:quantity, :price] })
 ```
 
-Scientific notation (e.g. `"1.5e3"`, `"6.022e23"`) is recognized and converted too. Bare-dot forms like `".5"` and `"3."` are left as Strings (they are not valid numbers here). Integers and floats convert identically on the C-accelerated and pure-Ruby paths.
+The Hash form requires exactly one of `only:`/`except:` with field name(s) — anything else (empty hash, unknown keys, both keys, empty lists, `nil`/boolean values) raises a `ValidationError` *(1.19.0+)*.
+
+Exponent forms (e.g. `"1e3"`, `"12E5"`, `"1.5e3"`) are NOT converted — they stay Strings *(changed in 1.19.0; only 1.18.x converted them)*. In real-world CSV data such values are far more often identifiers (short codes, hex IDs) than scientific notation, and auto-converting them corrupts data — e.g. an ID like `"0047583311587E590003"` became `Infinity`. If a column really does contain scientific notation, convert it per-column with [`value_converters`](./value_converters.md). Bare-dot forms like `".5"` and `"3."` are left as Strings (they are not valid numbers here). Integers and floats convert identically on the C-accelerated and pure-Ruby paths.
 
 ---
 
 ## `decimal_precision`
 
 **Default: `:auto`**
 
-Controls how decimal values (those with a `.` or an exponent) are converted. Integers are unaffected — they are always returned as `Integer`.
+Controls how decimal values (those with a `.`) are converted. Integers are unaffected — they are always returned as `Integer`.
 
 | Value         | Result                                                                                  |
 |---------------|-----------------------------------------------------------------------------------------|

diff --git a/docs/header_transformations.md b/docs/header_transformations.md
@@ -56,7 +56,9 @@ comment_regexp ──► strip_chars_from_headers ──► split on col_sep
 | 9 | `strings_as_keys` | `false` | Converts headers to symbols (skipped if `true` or `keep_original_headers`) |
 | 10 | `key_mapping` | `nil` | Renames or drops headers; use post-transformation key names as input |
 
-> `user_provided_headers` bypasses all file header reading and transformation entirely — your array is used as-is. Versions >1.13 automatically set `headers_in_file: false` when `user_provided_headers` is given; if the file has a header row you want to skip, set `headers_in_file: true` explicitly.
+> `user_provided_headers` bypasses all file header reading and transformation entirely — your array is used as-is; a `nil` entry drops that column. Versions >1.13 automatically set `headers_in_file: false` when `user_provided_headers` is given; if the file has a header row you want to skip, set `headers_in_file: true` explicitly.
+
+A quoted header containing an embedded newline is stitched across physical lines like a data row *(1.19.0+)* — the newline becomes `_` via the standard transformations (`"first\nname"` → `:first_name`).
 
 See [Configuration Options](./options.md) for full option reference.
 

diff --git a/docs/migrating_from_csv.md b/docs/migrating_from_csv.md
@@ -223,7 +223,7 @@ rows = SmarterCSV.process('sample.csv',
   convert_values_to_numeric: { except: [:zip_code, :phone, :account_number] })
 ```
 
-**High-precision decimals — scientific data and geo coordinates.** GPS/geo coordinates, scientific measurements, and financial figures routinely carry 16+ significant digits, where Ruby's `Float()`-based conversion (`converters: :numeric` / `:float`) silently rounds the value. SmarterCSV's default `decimal_precision: :auto` returns a `BigDecimal` once a value exceeds 16 significant digits (and a `Float` otherwise), so the full value is preserved; scientific notation (`6.022e23`, `1.6e-19`) is recognized as numeric too.
+**High-precision decimals — scientific data and geo coordinates.** GPS/geo coordinates, scientific measurements, and financial figures routinely carry 16+ significant digits, where Ruby's `Float()`-based conversion (`converters: :numeric` / `:float`) silently rounds the value. SmarterCSV's default `decimal_precision: :auto` returns a `BigDecimal` once a value exceeds 16 significant digits (and a `Float` otherwise), so the full value is preserved. (Exponent forms like `6.022e23` are not auto-converted — in CSV data they are usually identifiers, not numbers; use `value_converters` for columns that really contain scientific notation.)
 
 **With Ruby CSV (precision lost):**
 ```ruby

diff --git a/docs/options.md b/docs/options.md
@@ -33,7 +33,7 @@
 |--------|---------|-------------|
 | `:row_sep` | `$/` | Separates rows. Defaults to your OS row separator: `\n` on UNIX, `\r\n` on Windows. |
 | `:col_sep` | `","` | Separates each value in a row. |
-| `:quote_char` | `'"'` | Character used to quote CSV fields. |
+| `:quote_char` | `'"'` | Character used to quote CSV fields. Must be a single byte. |
 | `:force_quotes` | `false` | Forces each individual value to be quoted. |
 | `:headers` | `[]` | List of keys from the input to use as headers in the CSV file. ⚠️ Disables automatic header detection! |
 | `:map_headers` | `{}` | Like `:headers`, but also maps each key to a user-specified header value. ⚠️ Disables automatic header detection! |
@@ -121,7 +121,7 @@ See [Parsing Strategy](./parsing_strategy.md) for full details on quote handling
 | Option | Default | Explanation |
 |--------|---------|-------------|
 | `:strip_whitespace` | `true` | Remove whitespace before/after values and headers. |
-| `:convert_values_to_numeric` | `true` | Convert strings containing integers or floats (including scientific notation like `1.5e3`) to the appropriate numeric type. Accepts `{except: [:key1, :key2]}` or `{only: :key3}` to limit which columns. |
+| `:convert_values_to_numeric` | `true` | Convert strings containing integers or floats to the appropriate numeric type. Exponent forms like `1.5e3` or `12E5` stay Strings (1.19.0+) — they are usually identifiers, not numbers. Accepts `{except: [:key1, :key2]}` or `{only: :key3}` to limit which columns. |
 | `:decimal_precision` | `:auto` | How decimals are converted: `:auto` returns `Float` but `BigDecimal` above 16 significant digits (no precision loss); `:float` always returns `Float`; `:bigdecimal` always returns `BigDecimal`. Integers are unaffected. |
 | `:value_converters` | `nil` | Hash of `:header => converter`; converter can be a lambda/Proc or a class implementing `self.convert(value)`. See [Value Converters](./value_converters.md). |
 | `:remove_empty_values` | `true` | Remove key/value pairs where the value is `nil`, empty, or whitespace-only — any Unicode whitespace, same as Ruby's `String#blank?`. |
@@ -138,7 +138,7 @@ See [Bad Row Quarantine](./bad_row_quarantine.md) for full details.
 | `:on_bad_row` | `:raise` | Behavior when a row raises a parse error. `:raise` (default): re-raise, stopping processing. `:skip`: skip the bad row and continue. `:collect`: skip and append an error record to `reader.errors[:bad_rows]`. callable: called with the error record per bad row; processing continues. |
 | `:collect_raw_lines` | `true` | When collecting bad rows, include the raw stitched line in the error record. |
 | `:bad_row_limit` | `nil` | If set, raises `SmarterCSV::TooManyBadRows` after this many bad rows. |
-| `:field_size_limit` | `nil` | Maximum size of any extracted field in bytes. `nil` means no limit. Raises `SmarterCSV::FieldSizeLimitExceeded` (handled by `on_bad_row`) if a field or accumulating multiline buffer exceeds this size. Prevents DoS from runaway quoted fields or huge inline payloads. See [Bad Row Quarantine](./bad_row_quarantine.md#limiting-field-size-field_size_limit). |
+| `:field_size_limit` | `nil` | Maximum size of any extracted field in bytes. `nil` means no limit; the minimum allowed value is `4096` (it is overrun protection, not per-field validation). Raises `SmarterCSV::FieldSizeLimitExceeded` (handled by `on_bad_row`) if a field or accumulating multiline buffer exceeds this size. Prevents DoS from runaway quoted fields or huge inline payloads. See [Bad Row Quarantine](./bad_row_quarantine.md#limiting-field-size-field_size_limit). |
 
 ### Output & Diagnostics
 

diff --git a/docs/real_world_csv.md b/docs/real_world_csv.md
@@ -51,6 +51,7 @@ Real-world files come from dozens of different systems, each with their own defa
 | Windows-1252 / Latin-1 | 🔘 | Specify `file_encoding: 'windows-1252'`. Common in European financial exports, older SAP systems, QuickBooks. |
 | UTF-16 LE with BOM | 🔘 | Specify `file_encoding: 'utf-16le'`. Some Microsoft SQL Server and Access exports default to this. |
 | Shift-JIS / EUC-JP | 🔘 | Specify `file_encoding: 'shift_jis'` or `'euc-jp'`. Japanese ERP and POS systems. |
+| Invalid bytes / mislabeled encoding | ✅ | Never crashes — the affected field keeps its raw bytes exactly, recoverable via `force_encoding`. Opt into cleanup with `force_utf8` / `invalid_byte_sequence`. |
 
 ---