Skip to content
This repository was archived by the owner on Aug 12, 2026. It is now read-only.

Commit 6ab6391

Browse files
chaploudchaploud
committed
0020 proposed: generated namespaces; describe reads the artifact; 0019 review fixes
Three named parts, one file between them: 0020 (proposed, adversarial review in flight) picks up the shape 0015 accepted for later -- a generated .clj with the instance as an explicit first argument -- now that marshalling and host imports exist. Its part E lands here: the export walk moves out of instantiate into reflect-exports, and host/describe exposes it -- every export and signature from the compiled artifact alone, no instance, so the generator and instantiate can never disagree. Tested against echo: describe equals what the instance reports. An independent critique of c62a9e4 (16/20) found four things, all fixed: the engine docstring omitted the fifth resolution source (the system loader); a failed engine left its arena -- and therefore the bad library -- mapped for process life; lib-source was resolved twice from mutable sources, so a :symbol-missing report could name a library other than the loaded one; and the old-libwasmtime failure had no executable test. libjava now stands in for a pre-43 libwasmtime in the gate: loads fine, then :symbol-missing with the version floor in the message. Claude-Session: https://claude.ai/code/session_01XF5Hfq4Ca2N2XYEzWQQuHt
1 parent 88a128f commit 6ab6391

3 files changed

Lines changed: 245 additions & 50 deletions

File tree

‎doc/design/0020-generated-namespaces.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# 0020 — Generated namespaces: the ergonomics layer, as a file
2+
3+
**Status:** proposed · 2026-07-30 · the shape `0015` accepted for later, picked
4+
up now that its two blockers — marshalling and host imports — are gone.
5+
6+
## The question
7+
8+
`0014`'s primitive is honest but anonymous: `(i "local:zoo/shapes@0.3.0#take-flags")`
9+
gives an editor nothing to complete, clj-kondo nothing to check, and a reader
10+
no `:arglists`. `0015` established what the ergonomic layer must *not* be — a
11+
macro (clj-kondo cannot see through it, and the gate goes red outside `nix
12+
develop`), an ambient instance (laziness contaminates across instances
13+
whatever the mechanism), a `require` hook (hides the dependency from every
14+
tool that reads `ns` forms) — and named the survivor: **a generated `.clj`
15+
file, with the instance as an explicit argument.** This note decides what that
16+
file looks like.
17+
18+
## The decision
19+
20+
### A. One dev-time function, one readable file
21+
22+
```clojure
23+
(require '[cljwit.host.gen :as gen])
24+
(gen/write-ns! "resize.wasm" {:ns 'acme.resize :dir "src"})
25+
;; => "src/acme/resize.clj"
26+
```
27+
28+
`cljwit.host.gen` is a **separate namespace** from `cljwit.host`: generation
29+
is a dev-time act that reads a component's type, and requiring it in
30+
production buys nothing. The generated file is ordinary checked-in source —
31+
reviewed in diffs, seen by clj-kondo, clojure-lsp, and CIDER exactly because
32+
it is not clever. Its header names the source component, its sha-256, and the
33+
form that regenerates it, so drift is visible where drift is looked for: in
34+
version control.
35+
36+
### B. Every var takes the instance first
37+
38+
```clojure
39+
(defn thumbnail
40+
"WIT: thumbnail: func(img: list<u8>, w: u32, h: u32) -> list<u8>"
41+
[i img w h]
42+
((i "thumbnail") img w h))
43+
```
44+
45+
`0015` measured why ambience loses: `binding` conveys to exactly the paths
46+
that race (`future`, `pmap`) and a lazy seq realised under a second instance
47+
runs against the second silently. The explicit argument is not a compromise —
48+
it is what makes handing an instance to a `future` a visible act. Arity and
49+
names now exist statically: `(thumbnail i img)` is a clj-kondo warning and an
50+
`ArityException` naming `acme.resize/thumbnail`, not a gensym (`0015`'s
51+
macro-arity finding was about interned gensyms; a named `defn` does not have
52+
that problem).
53+
54+
### C. Names: the last WIT segment, and a collision is an error
55+
56+
The exact WIT string stays the runtime key inside the generated body — `0014`
57+
B is not renegotiated. The *var* takes the last segment, munged only where
58+
WIT's grammar demands it:
59+
60+
| WIT export | var |
61+
|---|---|
62+
| `plain` | `plain` |
63+
| `local:zoo/shapes@0.3.0#take-flags` | `take-flags` |
64+
| `…#[constructor]counter` | `counter` |
65+
| `…#[method]counter.bump` | `counter-bump` |
66+
| `…#[static]counter.reset` | `counter-reset` |
67+
68+
Two exports that munge to one name — two interfaces exporting `run`, say —
69+
**fail generation** with both WIT names in the message and a `:rename` option
70+
(`{:rename {"pkg:a/x@1.0.0#run" 'x-run}}`) as the exit. Silent suffixing
71+
(`run-2`) would bake an iteration order into an API; a namespace per
72+
interface multiplies files for a collision that `zoo.wit` — this repo's
73+
worst-case shape — does not have. Deterministic failure with a named exit
74+
follows `0014`'s nil-on-typo rule.
75+
76+
### D. `result` in return position gets `0012`'s sugar here
77+
78+
This is the layer `0012` deferred to: `(f i args)` returns the ok payload and
79+
throws `ex-info` with the lifted error under `:wit/error` (the shrunk
80+
contract — no type name); `(f* i args)` returns `[:ok v]` / `[:err e]`
81+
untouched. Two vars, generated together, only for exports whose *return* is a
82+
`result`. Everywhere else one var, no throwing — `0012`'s laziness argument
83+
stands.
84+
85+
### E. Reflection moves to the artifact, and becomes public
86+
87+
Generation needs exports and signatures without instantiating, and users have
88+
asked the same question of instances (`host/exports`, `host/signature`). The
89+
walk that `instantiate` does over the component type is extracted and exposed:
90+
91+
```clojure
92+
(host/describe artifact)
93+
;; => {"thumbnail" {:params [["img" {:kind :list :element :u8}] …]
94+
;; :result {:kind :list :element :u8}}, …}
95+
```
96+
97+
`describe` is useful on its own — a REPL user's first question about an alien
98+
`.wasm` — and it is the single source the generator consumes, so the
99+
generated file can never disagree with what `instantiate` will find.
100+
101+
## Why
102+
103+
- **Generated source is the only shape all three static tools see** —
104+
clj-kondo, clojure-lsp, the reader of a diff. `0015` measured the macro
105+
failing all three.
106+
- **The build step it adds is one the macro also paid**, per `0015`
107+
(`FileNotFoundException macroexpanding`), and here it is skippable: the
108+
file, once committed, needs no `.wasm` and no libwasmtime to load. `bb
109+
reflection` outside `nix develop` stays green.
110+
- **The instance argument is the honest concurrency story.** `0014` D's one
111+
store, one call rule is visible in the signature instead of hidden in a
112+
dynamic scope.
113+
114+
## Alternatives rejected
115+
116+
- **The macro, ambience, and the `require` hook** — rejected by `0015` on
117+
measurements; nothing here reopens them.
118+
- **Interning vars at instantiate** (`intern` from a plain function). Vars
119+
appear at run time, so clj-kondo and lsp are exactly as blind as with the
120+
macro, minus even the macro's syntactic marker.
121+
- **A namespace per WIT interface.** Buys collision-freedom `zoo.wit` does
122+
not need, at the cost of a file explosion and an import dance for the
123+
common one-interface world. Reopen if a real component's interface count
124+
makes the flat namespace unreadable.
125+
- **Silent collision suffixes.** An API whose names depend on reflection
126+
order breaks on the component author's next reorder.
127+
- **Emitting `defprotocol`/`definterface` for call-site speed.** `0013`:
128+
the binding is not worth optimising — a component call is ~800 ns; a map
129+
lookup in the fn body is noise.
130+
131+
## What would falsify this
132+
133+
- A WIT export name whose last segment is not a readable Clojure symbol
134+
after the munge table above — the generator refuses it today; a real
135+
component hitting that refusal reopens C.
136+
- A component whose regenerated file churns on every toolchain bump (sha in
137+
the header making diffs noisy) — would argue for dropping the sha line.
138+
- clj-kondo flagging the generated shape itself — would demand a header
139+
directive, and is checkable in this repo's own gate by generating against
140+
a fixture and linting the output.

‎src/cljwit/host.clj‎

Lines changed: 75 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -89,19 +89,21 @@
8989
;; DYLD_LIBRARY_PATH and the OS defaults.
9090
nm))))
9191

92-
(defn- open-lib ^SymbolLookup [^Arena arena explicit]
93-
(let [src (lib-source explicit)]
94-
(try
95-
(SymbolLookup/libraryLookup ^String src arena)
96-
(catch IllegalArgumentException e
97-
(throw (ex-info
98-
(str "libwasmtime failed to load from " (pr-str src)
99-
". Pass (engine {:lib path}), set -Dcljwit.wasmtime.lib"
100-
" or CLJWIT_WASMTIME_LIB, or install it where a package"
101-
" manager puts it (`brew install wasmtime`; wasmtime's"
102-
" *-c-api release tarballs carry it too). doc/design/0019")
103-
{:cljwit/error :no-library :cljwit/tried src}
104-
e))))))
92+
(defn- open-lib
93+
"Loads the already-resolved source — resolved once, by `api`, so a failure
94+
report can never name a different library than the one actually tried."
95+
^SymbolLookup [^Arena arena ^String src]
96+
(try
97+
(SymbolLookup/libraryLookup src arena)
98+
(catch IllegalArgumentException e
99+
(throw (ex-info
100+
(str "libwasmtime failed to load from " (pr-str src)
101+
". Pass (engine {:lib path}), set -Dcljwit.wasmtime.lib"
102+
" or CLJWIT_WASMTIME_LIB, or install it where a package"
103+
" manager puts it (`brew install wasmtime`; wasmtime's"
104+
" *-c-api release tarballs carry it too). doc/design/0019")
105+
{:cljwit/error :no-library :cljwit/tried src}
106+
e)))))
105107

106108
(defn- ffm
107109
"Binds one libwasmtime symbol. `ret` nil means void."
@@ -148,7 +150,7 @@
148150
[^Arena arena lib]
149151
(let [linker (Linker/nativeLinker)
150152
src (lib-source lib)
151-
lookup (open-lib arena lib)
153+
lookup (open-lib arena src)
152154
b (fn [nm ret & args] (ffm lookup linker src nm ret args))
153155
BOOL ValueLayout/JAVA_BOOLEAN]
154156
{:engine-new (b "wasm_engine_new" ADDR)
@@ -856,12 +858,18 @@
856858
857859
`:lib` names the libwasmtime to bind; without it the property
858860
`cljwit.wasmtime.lib`, then `CLJWIT_WASMTIME_LIB`, then the conventional
859-
install locations are tried, in that order (`0019`)."
861+
install locations, then the platform library name via the system loader
862+
are tried, in that order (`0019`)."
860863
(^Engine [] (engine {}))
861864
(^Engine [{:keys [lib]}]
862-
(let [arena (Arena/ofShared)
863-
a (api arena lib)]
864-
(->Engine arena a (invoke (:engine-new a)) (AtomicBoolean. false)))))
865+
(let [arena (Arena/ofShared)]
866+
;; If binding fails — no library, or one too old to have the component
867+
;; API — close the arena, or the bad library stays dlopen'd for the
868+
;; life of the process.
869+
(try
870+
(let [a (api arena lib)]
871+
(->Engine arena a (invoke (:engine-new a)) (AtomicBoolean. false)))
872+
(catch Throwable t (.close arena) (throw t))))))
865873

866874
(defn compile
867875
"Compiles a component. This is the expensive step — milliseconds, scaling
@@ -961,6 +969,39 @@
961969
(run! (fn [p] (invoke (:res-delete api) p)) ps)))
962970
(.set ^AtomicReference in-call nil))))))
963971

972+
(defn- reflect-exports
973+
"Every exported function, flattened: [{:key :path :sig} …]. A function
974+
inside an interface is keyed the way WIT spells it —
975+
\"pkg:name/iface@ver#func\" — and remembers the path it needs to resolve
976+
its export index (`0014` B)."
977+
[api ^Arena arena ^Engine e ^MemorySegment ct]
978+
(let [item-kind (fn [^MemorySegment it] (bit-and (long (.get it I8 (long 0))) 0xFF))
979+
walk (fn walk [ty nth-f count-f prefix]
980+
;; Fresh out-parameters per level: a recursive walk that shares
981+
;; them has the inner call overwrite the outer's name
982+
;; mid-iteration, which is a JVM crash, not a wrong answer.
983+
;; And `cat` here, not only at the top — a recursive producer
984+
;; has to flatten its own layer.
985+
(let [[pp lp] (name-pair arena)
986+
item ^MemorySegment (.allocate arena (long ITEM))
987+
one (fn [i]
988+
(when (invoke nth-f ty (.-ptr e) (long i) pp lp item)
989+
(let [nm (read-name pp lp)
990+
k (item-kind item)
991+
of (.get item ADDR (long ITEM-OF))]
992+
(cond
993+
(= ITEM-FUNC k)
994+
[{:key (if prefix (str prefix "#" nm) nm)
995+
:path (if prefix [prefix nm] [nm])
996+
:sig (reflect-func api arena of)}]
997+
998+
;; One level: WIT does not nest interfaces.
999+
(and (= ITEM-IFACE k) (nil? prefix))
1000+
(walk of (:iface-nth api) (:iface-count api) nm)))))]
1001+
(into [] (comp (keep one) cat)
1002+
(range (invoke count-f ty (.-ptr e))))))]
1003+
(walk ct (:export-nth api) (:export-count api) nil)))
1004+
9641005
(defn- reflect-imports
9651006
"What the component needs, keyed exactly as exports are (`0017` A)."
9661007
[api ^Arena arena ^Engine e ^MemorySegment ct]
@@ -1191,38 +1232,7 @@
11911232
;; calls into a store.
11921233
pending (atom [])
11931234
rt {:ctx ctx :api api :registry registry :pending pending}
1194-
item-kind (fn [^MemorySegment it] (bit-and (long (.get it I8 (long 0))) 0xFF))
1195-
;; Every export, flattened. A function inside an interface is keyed the
1196-
;; way WIT spells it — "pkg:name/iface@ver#func" — and remembers the
1197-
;; path it needs to resolve its export index (0014 B).
1198-
;; Fresh out-parameters per level: a recursive walk that shares them
1199-
;; has the inner call overwrite the outer's name mid-iteration, which
1200-
;; is a JVM crash rather than a wrong answer.
1201-
walk (fn walk [ty nth-f count-f prefix]
1202-
;; Fresh out-parameters per level: a recursive walk that shares
1203-
;; them has the inner call overwrite the outer's name
1204-
;; mid-iteration, which is a JVM crash, not a wrong answer.
1205-
;; And `cat` here, not only at the top — a recursive producer
1206-
;; has to flatten its own layer.
1207-
(let [[pp lp] (name-pair arena)
1208-
item ^MemorySegment (.allocate arena (long ITEM))
1209-
one (fn [i]
1210-
(when (invoke nth-f ty (.-ptr e) (long i) pp lp item)
1211-
(let [nm (read-name pp lp)
1212-
k (item-kind item)
1213-
of (.get item ADDR (long ITEM-OF))]
1214-
(cond
1215-
(= ITEM-FUNC k)
1216-
[{:key (if prefix (str prefix "#" nm) nm)
1217-
:path (if prefix [prefix nm] [nm])
1218-
:sig (reflect-func api arena of)}]
1219-
1220-
;; One level: WIT does not nest interfaces.
1221-
(and (= ITEM-IFACE k) (nil? prefix))
1222-
(walk of (:iface-nth api) (:iface-count api) nm)))))]
1223-
(into [] (comp (keep one) cat)
1224-
(range (invoke count-f ty (.-ptr e))))))
1225-
found (walk ct (:export-nth api) (:export-count api) nil)
1235+
found (reflect-exports api arena e ct)
12261236
sigs (into {} (map (fn [x] [(:key x) (:sig x)])) found)
12271237
index-of (fn [path]
12281238
(when (empty? path)
@@ -1263,6 +1273,21 @@
12631273
found)]
12641274
(->Instance art arena fns (alias-map fns) sigs store closed in-call registry dead rtab))))
12651275

1276+
(defn describe
1277+
"Every exported function and its declared shape, keyed by exact WIT name,
1278+
read from the compiled artifact alone — no instance needed. The same walk
1279+
`instantiate` performs, so the two can never disagree (`0020` E)."
1280+
[^Artifact art]
1281+
(closed! (.-closed art) "artifact")
1282+
(let [^Engine e (.-engine art)
1283+
_ (closed! (.-closed e) "engine")
1284+
api (.-api e)]
1285+
(with-open [arena (Arena/ofConfined)]
1286+
(let [ct (invoke (:comp-type api) (.-ptr art))]
1287+
(into {}
1288+
(map (fn [{:keys [key sig]}] [key sig]))
1289+
(reflect-exports api arena e ct))))))
1290+
12661291
(defn drop-failures
12671292
"Anything a `:drop` threw, and clears them. `0018` E: a destructor that
12681293
throws must not kill the instance, but a failure nobody can read is the

‎test/cljwit/host_test.clj‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,17 @@
7070
(host/engine {:lib "/nonexistent/libwasmtime.dylib"})))]
7171
(is (= :no-library (:cljwit/error (ex-data ex))))
7272
(is (re-find #"/nonexistent/libwasmtime\.dylib" (ex-message ex)))))
73+
(testing "a library that loads but is not wasmtime names the missing symbol"
74+
;; libjava ships with every JVM and dlopens fine, standing in for a
75+
;; pre-43 libwasmtime: loaded, then missing the component API.
76+
(let [jvm-lib (io/file (System/getProperty "java.home") "lib"
77+
(System/mapLibraryName "java"))]
78+
(if-not (.exists jvm-lib)
79+
(println "no libjava at" (str jvm-lib) "— skipping symbol-missing case")
80+
(let [ex (is (thrown? clojure.lang.ExceptionInfo
81+
(host/engine {:lib (str jvm-lib)})))]
82+
(is (= :symbol-missing (:cljwit/error (ex-data ex))))
83+
(is (re-find #">= 43" (ex-message ex)))))))
7384
(when lib
7485
(testing "the system property is honored — and beats the environment"
7586
(try
@@ -130,6 +141,25 @@
130141
(testing "many sequential calls — the in-call flag must be released each time"
131142
(is (= 100 (count (distinct (map (fn [n] ((:echo-s32 i) n)) (range 100))))))))))
132143

144+
(deftest describe-reads-the-artifact
145+
;; `0020` E: the generator's single source, and a REPL user's first
146+
;; question about an alien .wasm — no instance needed.
147+
(if-not lib
148+
(println "CLJWIT_WASMTIME_LIB unset — skipping describe test")
149+
(let [c (build-component!)]
150+
(try
151+
(with-open [e (host/engine)]
152+
(with-open [a (host/compile e (io/file c))]
153+
(let [d (host/describe a)]
154+
(is (= {:params [["v" :string]] :result :string}
155+
(get d "echo-string")))
156+
(testing "identical to what instantiate finds"
157+
(with-open [i (host/instantiate a)]
158+
(is (= d (into {}
159+
(map (fn [k] [k (host/signature i k)]))
160+
(host/exports i)))))))))
161+
(finally (.delete ^File c))))))
162+
133163
(deftest lifetimes-are-explicit
134164
(if-not lib
135165
(println "CLJWIT_WASMTIME_LIB unset — skipping host lifetime test")

0 commit comments

Comments
 (0)