Fix non-BMP text round-tripping by replacing <codecvt> (#25)

std::codecvt_utf8<wchar_t> transcodes one code unit per code point, so where
wchar_t is 16 bits it mishandled everything above U+FFFF in both directions, and
silently:

  - encoding emitted CESU-8, each surrogate half encoded as its own three-byte
    sequence (U+1F600 became ED A0 BD ED B8 80 instead of F0 9F 98 80);
  - decoding truncated the code point to its low 16 bits (U+1F600 became U+F600,
    and U+10000 became NUL).

Nothing threw, so text was corrupted without any indication. The same code was
correct where wchar_t is 32 bits, so identical data behaved differently per
platform. <codecvt> is additionally deprecated since C++17 with no standard
replacement, and this library's baseline is C++11.

Conversion now pivots on Unicode code points in internal/utf8.h, and the
wchar_t adapters in string_utilities.cc select the UTF-16 or UTF-32 path by the
width of wchar_t. The public API is unchanged: text fields remain std::wstring.

Malformed input is rejected with std::range_error rather than being altered,
matching what std::wstring_convert did for ill-formed sequences. Rejected are
overlong encodings, truncated sequences, invalid lead and continuation bytes,
surrogates encoded directly in UTF-8, unpaired UTF-16 surrogates, and anything
above U+10FFFF.

The code-point core is tested directly, so the UTF-16 surrogate path - the one
that was broken, and which is unreachable through the wstring API on Linux and
macOS - is covered on every platform rather than only on Windows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: jkalias

README.md

@@ -136,6 +136,17 @@ Supported field macros:
 | `sqlite_reflection::TimePoint` | `MEMBER_DATETIME(name)` |
 | member function declaration | `FUNC(signature)` |
 
+### Text encoding
+
+`MEMBER_TEXT` fields are `std::wstring` in C++ and are stored as UTF-8 in SQLite. The full
+Unicode range is supported, including code points above U+FFFF such as emoji and the CJK
+extension blocks, and text round-trips unchanged on every supported platform regardless of
+whether `wchar_t` is 16 bits (Windows) or 32 bits (Linux, macOS).
+
+Text that is not valid Unicode is rejected rather than silently altered: converting a
+`std::wstring` containing unpaired UTF-16 surrogates, or reading a column whose bytes are not
+well-formed UTF-8, throws `std::range_error`.
+
 ### Nullable fields
 
 Each field macro has a `_NULLABLE` variant (`MEMBER_INT_NULLABLE`, `MEMBER_REAL_NULLABLE`,

src/internal/utf8.h

@@ -0,0 +1,63 @@
+// MIT License
+//
+// Copyright (c) 2026 Ioannis Kaliakatsos
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+#include <cstddef>
+#include <string>
+
+#include "reflection_export.h"
+
+namespace sqlite_reflection {
+/// Unicode transcoding that does not depend on the platform width of wchar_t.
+///
+/// The library previously converted text with std::wstring_convert and
+/// std::codecvt_utf8<wchar_t>. That facet transcodes UTF-8 to and from a single code unit per
+/// code point, so where wchar_t is 16 bits (Windows) it silently mishandled everything above
+/// U+FFFF: encoding produced CESU-8 (each surrogate half encoded separately, U+1F600 became
+/// ED A0 BD ED B8 80 rather than F0 9F 98 80) and decoding truncated the code point to its low
+/// 16 bits (U+1F600 became U+F600, U+10000 became NUL). The same code was correct where
+/// wchar_t is 32 bits, so identical data behaved differently per platform. <codecvt> is also
+/// deprecated since C++17 with no standard replacement, and this library's baseline is C++11.
+///
+/// Conversion therefore pivots on Unicode code points here, and the wchar_t adapters in
+/// StringUtilities select the UTF-16 or UTF-32 path by the width of wchar_t.
+///
+/// Every function validates its input and throws std::range_error on anything malformed -
+/// matching the behavior std::wstring_convert had for ill-formed sequences. Rejected are
+/// overlong encodings, truncated sequences, stray or invalid lead and continuation bytes,
+/// surrogate code points encoded directly (CESU-8), unpaired UTF-16 surrogates, and anything
+/// above U+10FFFF.
+class REFLECTION_EXPORT Utf8 {
+public:
+    /// Encodes Unicode code points as UTF-8
+    static std::string FromCodePoints(const char32_t* code_points, size_t count);
+
+    /// Decodes UTF-8 into Unicode code points
+    static std::u32string ToCodePoints(const char* utf8_string, size_t byte_count);
+
+    /// Encodes Unicode code points as UTF-16 code units, using a surrogate pair above U+FFFF
+    static std::u16string Utf16FromCodePoints(const char32_t* code_points, size_t count);
+
+    /// Decodes UTF-16 code units, combining surrogate pairs, into Unicode code points
+    static std::u32string CodePointsFromUtf16(const char16_t* utf16_string, size_t count);
+};
+}  // namespace sqlite_reflection

src/string_utilities.cc

@@ -22,14 +22,66 @@
 
 #include "internal/string_utilities.h"
 
-#include <codecvt>
 #include <numeric>
-#ifndef _WIN32
-#include <locale>
-#endif
+
+#include "internal/utf8.h"
 
 using namespace sqlite_reflection;
 
+namespace {
+// wchar_t holds UTF-16 code units where it is 16 bits wide (Windows) and Unicode code points
+// where it is 32 bits wide (Linux, macOS). Selecting the path by width is what keeps text above
+// U+FFFF identical across platforms; a single facet cannot serve both, which is the defect this
+// replaced. See internal/utf8.h.
+template <size_t WideCharSize>
+struct WideCodec;
+
+template <>
+struct WideCodec<2> {
+    static std::u32string ToCodePoints(const std::wstring& wide_string) {
+        std::u16string code_units;
+        code_units.reserve(wide_string.size());
+        for (size_t i = 0; i < wide_string.size(); ++i) {
+            code_units.push_back(static_cast<char16_t>(wide_string[i]));
+        }
+        return Utf8::CodePointsFromUtf16(code_units.data(), code_units.size());
+    }
+
+    static std::wstring FromCodePoints(const std::u32string& code_points) {
+        const auto code_units = Utf8::Utf16FromCodePoints(code_points.data(), code_points.size());
+        std::wstring wide_string;
+        wide_string.reserve(code_units.size());
+        for (size_t i = 0; i < code_units.size(); ++i) {
+            wide_string.push_back(static_cast<wchar_t>(code_units[i]));
+        }
+        return wide_string;
+    }
+};
+
+template <>
+struct WideCodec<4> {
+    static std::u32string ToCodePoints(const std::wstring& wide_string) {
+        std::u32string code_points;
+        code_points.reserve(wide_string.size());
+        for (size_t i = 0; i < wide_string.size(); ++i) {
+            code_points.push_back(static_cast<char32_t>(wide_string[i]));
+        }
+        return code_points;
+    }
+
+    static std::wstring FromCodePoints(const std::u32string& code_points) {
+        std::wstring wide_string;
+        wide_string.reserve(code_points.size());
+        for (size_t i = 0; i < code_points.size(); ++i) {
+            wide_string.push_back(static_cast<wchar_t>(code_points[i]));
+        }
+        return wide_string;
+    }
+};
+
+typedef WideCodec<sizeof(wchar_t)> PlatformWideCodec;
+}  // namespace
+
 std::string StringUtilities::FromInt(int64_t value) {
     return std::to_string(value);
 }
@@ -45,15 +97,13 @@ std::string StringUtilities::FromDouble(double value) {
 }
 
 std::string StringUtilities::ToUtf8(const std::wstring& wide_string) {
-    std::wstring_convert<std::codecvt_utf8<wchar_t>> converter;
-    auto utf8_string = converter.to_bytes(wide_string.data(), wide_string.data() + wide_string.size());
-    return utf8_string;
+    const auto code_points = PlatformWideCodec::ToCodePoints(wide_string);
+    return Utf8::FromCodePoints(code_points.data(), code_points.size());
 }
 
 std::wstring StringUtilities::FromUtf8(const char* utf8_string, size_t byte_count) {
-    std::wstring_convert<std::codecvt_utf8<wchar_t>> converter;
-    auto wide_string = converter.from_bytes(utf8_string, utf8_string + byte_count);
-    return wide_string;
+    const auto code_points = Utf8::ToCodePoints(utf8_string, byte_count);
+    return PlatformWideCodec::FromCodePoints(code_points);
 }
 
 std::string StringUtilities::Join(const std::vector<std::string>& list, const std::string& separator) {