Skip to content

Commit 6477a06

Browse files
authored
Merge pull request #57 from BuddhiLW/docs/string-code-point-divergence
docs(strings): record code-point string indexing as AD-076 (#1)
2 parents 11ea603 + de0ac85 commit 6477a06

4 files changed

Lines changed: 127 additions & 0 deletions

File tree

‎.dev/accepted_divergences.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1190,3 +1190,32 @@ accepted:
11901190
why: 'Timestamp extends Date on the JVM, so Date.equals(Object) accepts a Timestamp by millisecond, while Timestamp.equals(Object) rejects every non-Timestamp. The Timestamp javadoc itself calls the pair not symmetric and says the inheritance is implementation inheritance, not type inheritance. Porting it would make `=` depend on argument order, and every hashed collection would then depend on insertion order: clj builds (hash-set t d) as one element and (hash-set d t) as two. cljw models Timestamp as its own value type (D-382: a distinct descriptor carrying epoch-ms plus nanos, never collapsed onto Date), and `=` over temporal values compares within one type, which keeps `=` symmetric and `=` implies equal hash, the two laws src/testing/prop_equal.zig pins. The hashes already agree with clj, so only the cross-type Date-first comparison differs.'
11911191
derives_from: 'D-382 (Timestamp is its own neutral value type with a distinct descriptor; collapsing it onto Date was the rejected smell, user 2026-06-10) + F-014 (linguistically-general Java, not a JVM reimplementation: the JDK equals-contract violation is not ported) + the symmetric `=` law pinned by src/testing/prop_equal.zig.'
11921192
pin: test/e2e/temporal_hash_eq.sh same_hash_distinct_types - asserts on both backends that (= d t) is false and (get {t :t} d) is nil for a Date and a Timestamp of the same millisecond, next to the clj-matching (= t d) false and (count (hash-set d t)) 2.
1193+
- id: AD-076
1194+
area: strings
1195+
summary: 'A string is indexed by Unicode CODE POINT, not by UTF-16 code unit. `count`, `subs`, `nth`, `get`, `seq`/`first`, `(map int s)` and the interop methods `.length`, `.charAt`, `.indexOf` and `.substring` all count a character outside the Basic Multilingual Plane (emoji, rare CJK, most historic scripts) as ONE position, where clj counts its surrogate pair as two. BMP text, including multibyte characters such as é or 世, indexes identically in both runtimes.'
1196+
example: '(count "🌍") => cljw 1 / clj 2. (count "ab🌍") => cljw 3 / clj 4. (.length "🌍") => cljw 1 / clj 2. (subs "a🌍b" 1 2) => cljw "🌍" / clj a lone high surrogate. (vec (map int "🌍")) => cljw [127757] / clj [55356 57101]. (str (first "🌍")) => cljw "🌍" / clj a lone high surrogate. (.indexOf "a🌍b" "b") => cljw 2 / clj 3. Measured on cljw v1.14.11; clj 1.12 column from GitHub issue #1.'
1197+
why: |-
1198+
UTF-16 is how the JVM happens to store a String, not part of Clojure's
1199+
string semantics: clj's own `count` docstring says "number of items in
1200+
the collection", and a surrogate half is not an item any user wrote.
1201+
cljw stores strings as UTF-8 and has no UTF-16 buffer to index, so
1202+
matching clj would mean synthesising surrogate halves on every index
1203+
operation: values that cannot be printed, that are not valid Unicode
1204+
scalar values, and whose only observable use is to be glued back
1205+
together. `(subs s i j)` handing back half a pair is a bug generator in
1206+
clj; in cljw every index lands on a character boundary, so no string
1207+
operation can produce invalid text.
1208+
The interop methods follow the same model on purpose. A second,
1209+
UTF-16-unit view behind `.length` / `.charAt` / `.indexOf` /
1210+
`.substring` would make `(= (count s) (.length s))` false for the same
1211+
string and would mix two index spaces in one program: an index from
1212+
`.indexOf` fed to `subs` would land on the wrong character. One index
1213+
space for every string operation is the narrower and more defensible
1214+
claim; the interop names are kept for source compatibility, and the
1215+
ledger records that they are code-point based.
1216+
Ported JVM code is affected only where it does arithmetic on indices
1217+
against outside UTF-16 data (a JavaScript or Java peer's offsets, a
1218+
protocol counting UTF-16 units). docs/clojure_vs_clojurewasm.md gives
1219+
the porting guidance.
1220+
derives_from: 'F-011 read at the VALUE level (a string is a sequence of characters; the JVM storage encoding is not surface), plus D-217 (a String is Indexed by codepoint) and D-179 / O-060 (string seq is a UTF-8 byte-offset view over code points). No JVM, ADR-0059: there is no UTF-16 backing store to expose.'
1221+
pin: test/clj/suites/string_code_point_test.clj (every expression in the example, language and interop surfaces) + test/golden/cases/string_unicode.clj (count of a string holding 🌍, and (count "🌍") => 1).

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,14 @@ first stable `1.0.0` tag; pre-1.0 `alpha` / `rc` tags may still change surfaces.
4343
`(wasm/clear-cache!)` drops every cached module and returns how many there
4444
were.
4545

46+
- **Strings indexed by code point are now a recorded divergence (AD-076).**
47+
Behaviour is unchanged: `count`, `subs`, `nth`, `seq` and the interop
48+
methods `.length`, `.charAt`, `.indexOf` and `.substring` count a character
49+
outside the BMP as one position, where clj counts its UTF-16 surrogate pair
50+
as two (`(count "🌍")` is 1 in cljw, 2 in clj). The rationale and porting
51+
guidance are in `docs/clojure_vs_clojurewasm.md`, and
52+
`test/clj/suites/string_code_point_test.clj` pins the answers (#1).
53+
4654
### Fixed
4755

4856
- **A number token runs to the next delimiter and is rejected whole, as in

‎docs/clojure_vs_clojurewasm.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,49 @@ STM's committed state is identical (4 threads × 100 `(dosync (alter c inc))`
118118
differs. Locking and `with-local-vars` are clj-faithful *inside* their normal
119119
use; only the literal-as-lock and escaped-var anti-patterns diverge.
120120

121+
### Strings are indexed by code point, not UTF-16 unit
122+
123+
cljw stores a string as UTF-8 and every index into it counts Unicode code
124+
points. The JVM stores UTF-16 and clj indexes by code unit, so a character
125+
outside the Basic Multilingual Plane (emoji, rare CJK, most historic scripts)
126+
is one position in cljw and two (a surrogate pair) in clj. BMP text, including
127+
multibyte characters such as `é` or `世`, indexes the same in both.
128+
129+
| Expression | Clojure 1.12 (JVM) | ClojureWasm | AD |
130+
|----------------------------|-------------------------------|---------------|--------|
131+
| `(count "🌍")` | `2` | `1` | AD-076 |
132+
| `(count "ab🌍")` | `4` | `3` | AD-076 |
133+
| `(.length "🌍")` | `2` | `1` | AD-076 |
134+
| `(subs "a🌍b" 1 2)` | a lone high surrogate | `"🌍"` | AD-076 |
135+
| `(vec (map int "🌍"))` | `[55356 57101]` | `[127757]` | AD-076 |
136+
| `(str (first "🌍"))` | a lone high surrogate | `"🌍"` | AD-076 |
137+
| `(.indexOf "a🌍b" "b")` | `3` | `2` | AD-076 |
138+
139+
cljw column measured on v1.14.11; clj column from
140+
[issue #1](https://github.com/BuddhiLW/ClojureWasm/issues/1).
141+
142+
**Both surfaces use code points, on purpose.** The language surface (`count`,
143+
`subs`, `nth`, `get`, `seq`) and the Java interop names (`.length`, `.charAt`,
144+
`.indexOf`, `.substring`) share one index space, so `(= (count s) (.length s))`
145+
always holds and an index from `.indexOf` is safe to pass to `subs`. The
146+
interop methods are therefore *not* a faithful `java.lang.String#length`: they
147+
keep the Java names for source compatibility and answer in code points.
148+
149+
**Porting JVM code.** Code that only takes indices from one string operation
150+
and feeds them to another (`.indexOf` then `subs`, `count` then a loop over
151+
`nth`) keeps working unchanged, and stops producing half surrogate pairs.
152+
Look again at code that:
153+
154+
- compares a length or offset with one computed **outside** cljw: a UTF-16
155+
offset from a JavaScript or Java peer, an LSP `character` position, a
156+
protocol field counting UTF-16 units. Convert at the boundary rather than
157+
inside the program.
158+
- tests `(Character/isHighSurrogate c)` / `isLowSurrogate`, or reassembles
159+
pairs with `Character/toCodePoint`: in cljw a char from a string is already a
160+
whole code point, so those branches never fire.
161+
- sizes a buffer from `.length` expecting bytes. Neither runtime gives bytes
162+
there; use `(count (.getBytes s "UTF-8"))`, which agrees in both.
163+
121164
### Reader & security
122165

123166
| Behaviour | Clojure (JVM) | ClojureWasm | AD |
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
;; AD-076: a string is a sequence of Unicode CODE POINTS in cljw, where JVM
2+
;; Clojure exposes UTF-16 CODE UNITS. Every character outside the Basic
3+
;; Multilingual Plane (emoji, rare CJK, most historic scripts) is one position
4+
;; here and two (a surrogate pair) on the JVM.
5+
;;
6+
;; This suite PINS the cljw answers so the divergence cannot flip silently in
7+
;; either direction: a change that starts counting UTF-16 units must remove the
8+
;; AD-076 entry in .dev/accepted_divergences.yaml and rewrite these
9+
;; assertions, deliberately. The clj answer is on each line's comment,
10+
;; measured against JVM Clojure 1.12 (GitHub issue #1).
11+
;;
12+
;; The language surface (count, subs, nth, seq) and the interop surface
13+
;; (.length, .charAt, .indexOf, .substring) follow the SAME model: cljw has one
14+
;; string representation, UTF-8 indexed by code point, and does not keep a
15+
;; second UTF-16 index for the interop methods.
16+
(ns suites.string-code-point-test
17+
(:require [clojure.test :refer [deftest is testing]]))
18+
19+
(def globe "🌍") ; U+1F30D, a surrogate pair in UTF-16
20+
21+
(deftest count-is-in-code-points
22+
(is (= 1 (count globe))) ; clj 2
23+
(is (= 3 (count "ab🌍"))) ; clj 4
24+
(testing "a BMP multibyte character is one position in both runtimes"
25+
(is (= 5 (count "héllo")))))
26+
27+
(deftest subs-and-nth-never-split-a-pair
28+
(is (= "🌍" (subs "a🌍b" 1 2))) ; clj a lone high surrogate
29+
(is (= "b" (subs "a🌍b" 2 3))) ; clj the low surrogate
30+
(is (= \b (nth "a🌍b" 2)))) ; clj the low surrogate
31+
32+
(deftest a-char-is-a-code-point
33+
(is (= [127757] (vec (map int globe)))) ; clj [55356 57101]
34+
(is (= "🌍" (str (first globe)))) ; clj a lone high surrogate
35+
(is (char? (first globe))))
36+
37+
(deftest interop-methods-follow-the-same-model
38+
(is (= 1 (.length globe))) ; clj 2
39+
(is (= \🌍 (.charAt "a🌍b" 1))) ; clj \? (high surrogate)
40+
(is (= 2 (.indexOf "a🌍b" "b"))) ; clj 3
41+
(is (= "🌍" (.substring "a🌍b" 1 2)))) ; clj a lone high surrogate
42+
43+
(deftest where-the-two-runtimes-agree
44+
(testing "UTF-8 byte length is a byte count, not an index, so it matches"
45+
(is (= 4 (count (.getBytes globe "UTF-8")))))
46+
(testing "reverse keeps the pair whole in both (StringBuilder.reverse does too)"
47+
(is (= "b🌍a" (clojure.string/reverse "a🌍b")))))

0 commit comments

Comments
 (0)