Version 1.19.0

[tilo]

Reverted Behavior Changes

• Exponent forms are no longer auto-converted to numbers (#345).

Version 1.18.0 started converting scientific notation ("1e3", "12E5", "1.5e3") to Floats. In real-world CSV data, digits-E-digits values are far more often identifiers (short codes, hex IDs) than scientific notation, and the auto-conversion corrupted them irreversibly — an ID like "0047583311587E590003" came back as Infinity.

As of 1.19.0, exponent forms always stay Strings — as they did in every version before 1.18.0. If a column really does contain scientific notation, convert it per-column with value_converters:

SmarterCSV.process(file, value_converters: { measurement: ->(v) { v.to_f } })

Thanks to Denis Sadomowski for the report.

Bug Fixes

• A partial multi-char separator at end-of-line is no longer consumed as a separator on the C path (C/Ruby parity — silent data loss). With col_sep: '||', a value or header ending in a lone | lost that character ("y|" came back as "y"). The separator comparison (and the close-quote lookahead) is now bounded by the end of the line, which also removes an out-of-bounds read for multi-char separators near end-of-line.

• An empty line now yields nil for ALL columns on the C path too (C/Ruby parity, remove_empty_values: false). The C path gave the first column an empty string ({a: "", b: nil, ...}) where the Ruby path — matching "".split — yields no fields, so every column is padded with nil.

• A nil entry in user_provided_headers now drops that column on the C path too (C/Ruby parity). The nil key survived into the row hashes on the accelerated path.

• Non-ASCII missing_header_prefix (e.g. "spalte_ä_") no longer raises EncodingError on the C path. Generated extra-column keys are now interned as UTF-8 symbols.

• The Hash form of convert_values_to_numeric is now validated and normalized. It requires exactly one of only:/except: with field name(s) (String/Symbol or an Array of them); an empty hash, unknown keys, both keys together, or nil/boolean values raise a ValidationError — previously the C and Ruby paths silently disagreed on these shapes (e.g. {} meant "convert nothing" on the C path and "convert everything" on the Ruby path). The listed names are normalized to the row-key type, so only:/except: now also works together with strings_as_keys / keep_original_headers (Symbol selectors silently matched nothing there before, on both paths).

• A row consisting only of NUL bytes now counts as blank on the C path too (C/Ruby parity). The blank-row test follows Ruby's value.strip.empty?, and String#strip also removes NUL bytes (\0) — the C path kept such rows with strip_whitespace: false. The per-field value is unchanged: with remove_empty_hashes: false a NUL byte is still kept as data on both paths.

• headers: { only: } now short-cuts on the pure-Ruby path too (C/Ruby parity + speed). The point of only: is to stop parsing each row right after the last wanted column — the C path did that; the Ruby path parsed every column, built the full row hash, and then deleted the unwanted keys, so it also discovered extra columns behind the last wanted one (reader.headers grew with :column_N entries the C path never saw) and raised MalformedCSV for an unclosed quote in an unwanted trailing column that the C path ignores. Both paths now stop identically after the last wanted column.

• field_size_limit now also catches oversized digit-only fields on the C path (C/Ruby parity). The C path converted a huge digit field to a number before the size check (which only measured Strings), so the limit never fired — and converting e.g. a 200KB digit string to an Integer is exactly the expensive overrun the option exists to prevent. The C parser now checks the raw field size before any conversion. Additionally, field_size_limit values below 4096 now raise a ValidationError — the option is overrun protection, not per-field validation.

• All empty field values are now ONE shared, frozen, UTF-8 empty-string object — on both paths. This was the C path's design (no per-empty-field object retained in the results), but the shared object was mutable — appending to one empty value silently changed every other empty value in the result — and the Ruby path allocated a fresh string per empty field. Mutating an empty value now raises FrozenError on both paths. Relevant with remove_empty_values: false; with the default true, empty values are removed anyway.

• Exotic option sizes fall back to the pure-Ruby parser instead of silently truncating. The C parse context stores col_sep (7 bytes), row_sep (15), and missing_header_prefix (63) in fixed-size buffers; longer values produced wrong results on the accelerated path. The reader now automatically uses the pure-Ruby parser for these, which handles any length.

• Writer: fields are now wrapped in the configured quote_char, not a hard-coded double quote. Output written with a custom quote_char (e.g. "'") could not be read back: the custom quote_char was doubled correctly inside the field, but the field itself was wrapped in ".

• Reader#each without a block no longer clears the configured chunk_size. Calling each in its Enumerator form (no block) overwrote options[:chunk_size] with nil, so a later each_chunk on the same Reader ignored the configured chunk size.

• C path now strips a stray trailing \r from values (C/Ruby parity). With strip_whitespace: true (the default), the C-accelerated path only stripped spaces and tabs, so a \r survived at the end of the last field on CRLF lines in mixed LF/CRLF files (and in CRLF files read with an explicit row_sep: "\n"). The C path now strips exactly Ruby's String#strip character set (space, \t, \n, \v, \f, \r, \0), matching the pure-Ruby path.

• nil_values_matching now matches the raw string value on the C path too (C/Ruby parity). The pattern is written against what's in the file, but the C-accelerated path converted values to numbers first — so a pattern like /\A007\z/ never matched (the matcher only ever saw 7). When nil_values_matching is set, the C parser now defers numeric conversion and zero-removal to the Ruby hash transformations, which apply the pattern to the raw string first — the same order as the pure-Ruby path.

• quote_char: :auto now raises a ValidationError. There is no auto-detection for quote_char (only for row_sep and col_sep), but validation accepted :auto and the Reader then crashed with a NoMethodError.

• Duplicate-header disambiguation no longer collides with a real column name. With headers name,name,name2, the second name was renamed to name2 (default suffix + counter), colliding with the real third column — and the reader then raised DuplicateHeaders, defeating the disambiguation feature. The counter is now bumped past taken names (the second name becomes name3).

• The user_provided_headers array is no longer mutated. When rows contained more columns than headers, the reader appended column_N entries directly into the caller's array (and into options[:user_provided_headers], so a reused options hash silently changed behavior on the next file). The reader now works on its own copy.

• headers: { only: } / { except: } now works together with strings_as_keys / keep_original_headers. The selector values were always normalized to Symbols, but in those modes the row keys are Strings — so nothing matched, and with headers: { only: } every row came back empty (then was dropped by remove_empty_hashes): silent total data loss. The selectors are now normalized to the row-key type.

• A quoted header containing an embedded newline is now stitched across physical lines, like data rows. Previously the first header fragment was silently lost and the second fragment was parsed as a data row — silent corruption. The embedded newline becomes _ via the standard header transformations ("first\nname" → :first_name); an unclosed quote that reaches end-of-file raises MalformedCSV.

• An empty-string header key is now dropped on the C path too (C/Ruby parity). With strings_as_keys: true and duplicate_header_suffix: nil (which disables the column_N auto-naming), an empty header produced a '' String key that the Ruby path dropped but the C path kept (its cleanup only deleted the :"" Symbol form).

• A one-character, multi-byte col_sep (e.g. 'é') no longer crashes the pure-Ruby parser (C/Ruby parity). The Ruby parser's byte-level fast path was gated on the separator's character count, then scanned for its first byte only — which also occurs as the lead byte of other characters — and raised ArgumentError on quoted lines. The fast path is now gated on bytesize; multi-byte separators take the character-level path, matching the C parser's results.

• The multiline stitch gate no longer fabricates MalformedCSV on rows the parser can close (pure-Ruby path). The gate (detect_multiline_strict) disagreed with the parser in three ways: it lacked the doubled-quote precedence rule ("" inside a quoted field, issue Escaped double quote followed by comma issue #334), it had no backslash-escape awareness (quote_escaping: :backslash, and the primary interpretation of :auto), and it walked the unchomped line, flipping end-of-line close decisions. The gate now models the parser's rules exactly, and under :auto reports "still open" only when both the backslash and RFC interpretations are open. A seeded differential fuzz spec (parity_fuzz_spec.rb) now guards C/Ruby parity permanently; 30,000 randomized inputs run clean on both paths.

• A trailing \r before an LF row separator is now treated as part of the line terminator on the C path too (C/Ruby parity). Ruby's String#chomp("\n") removes \r\n, \r, or \n; the C path chomped the row separator literally, so the surviving \r made a CRLF line whose last field is quoted raise MalformedCSV (even with default options), and with strip_whitespace: false values came back as "x\r" / "1\r" (String) instead of "x" / 1. Found by differential fuzzing.

• Multi-byte characters directly before a literal quote no longer crash the pure-Ruby parser (C/Ruby parity). An unquoted field like é"x made the Ruby parser's byte-level skip-ahead hand String#byteindex a mid-character byte offset — IndexError: offset does not land on character boundary. The skip-ahead now falls back to the byte loop when the scan position is mid-character. Found by differential fuzzing against the C path, which parsed these fine.

• Invalid bytes in the input no longer crash the pure-Ruby parser (C/Ruby parity). Typical case: Latin-1 data mislabeled as UTF-8. The C path parses leniently and preserves the field's raw bytes; the Ruby fallback raised ArgumentError ("invalid byte sequence in UTF-8") — which is not a SmarterCSV::Error, so even on_bad_row: :skip couldn't quarantine it. The Ruby parser now processes such lines at the byte level and re-tags the fields with the original encoding — bytes preserved exactly, never transcoded, so the data stays recoverable (e.g. via force_encoding('ISO-8859-1')). The value-transformation regexes (nil_values_matching, zero-removal, numeric conversion) skip invalid-encoding values instead of raising. Cleanup remains opt-in via force_utf8 / invalid_byte_sequence.

• nil_values_matching no longer switches off numeric conversion and zero-removal on the C path. (Fixes a regression introduced while making the option match raw strings, above: the C parser correctly deferred those transformations to Ruby, but the accelerated post-processing never ran them.) Non-matching values now get numeric conversion, zero-removal, and value_converters in the same order as the pure-Ruby path.

Tests

• Every parsing spec now runs on BOTH the C-accelerated and the pure-Ruby path via [true, false] acceleration loops (~420 additional examples), a seeded differential parity-fuzz spec (2,000 randomized inputs per run, including combined option sets) guards C/Ruby parity permanently, and line coverage is at 100%.

Test results

RSpec tests: 2,595 → 3,164 (+569 tests) — every parsing spec runs on BOTH the C-accelerated and the pure-Ruby path via [true, false] acceleration loops; a seeded differential parity-fuzz spec (2,000 randomized inputs per run, including combined option sets) guards C/Ruby parity permanently; line coverage is at 100% (1546/1546 lines). Verified clean with 30,000+ randomized fuzz inputs offline. Rubocop: no offenses in changed files.