Skip to content

Commit b92c2de

Browse files
authored
feat: add owned typed arrays (#68)
## Summary - add `js.OwnedTypedArray(Element, array_type)` and concrete `Owned*Array` aliases - transfer allocator-owned element storage to an external ArrayBuffer without copying - support owned typed arrays as return values from DSL functions and class methods - make ownership transfer transactional: pre-transfer failures preserve the Zig owner; successful or potentially transferred values leave the source empty - align `OwnedBuffer.intoValue` with the same pointer-based transactional ownership model while retaining its existing unsupported-runtime copy fallback - validate each `Element` against its N-API TypedArray kind at compile time - keep `TypedArray.fromExternal` unchanged from `main` - propagate `NoExternalBuffersAllowed` unchanged for owned typed arrays; no copy fallback ## API ```zig pub fn serialize() !js.OwnedUint8Array { const allocator = js.allocator(); const data = try allocator.alloc(u8, 32); // Fill data... return js.OwnedUint8Array.fromOwnedSlice(allocator, data); } ``` `fromOwnedSlice` takes an existing allocator-owned mutable slice without copying. `fromSlice` creates an owned copy when that is what the caller needs. `intoValue` takes `*OwnedTypedArray`. Once ownership transfers, it empties the source, so a normal deferred `deinit` is safe. The allocator must remain valid until the ArrayBuffer finalizer runs. `OwnedBuffer.intoValue` now follows the same pointer-based transfer contract. It retains its existing `NoExternalBuffersAllowed` copy fallback and consumes the source only after that copy succeeds. ## Lodestar-Z use case In the Lodestar-Z BLS bindings, `compress` and `serialize` currently produce fixed-size stack arrays and pass them to `TypedArray.fromExternal`, which copies them into allocator-owned native storage before creating the external ArrayBuffer. With this API, the bindings can allocate the output slice on the native heap, let `blst` write into it directly, and return it with `OwnedUint8Array.fromOwnedSlice`. This removes one stack-to-native-heap copy per result while preserving the external ArrayBuffer GC lifecycle. ## Ownership boundary - zero length: create a normal zero-length V8 ArrayBuffer and empty the source only after TypedArray creation succeeds - owner heap allocation failure: leave the allocation owned by the caller - `NoExternalBuffersAllowed`, `PendingException`, or `CannotRunJS`: Node returns before finalizer registration, so restore ownership to the caller and propagate the original error - other external ArrayBuffer errors: leave the source empty because Node may already have installed or invoked the finalizer - typed-array view creation failure or success: leave the source empty; the external ArrayBuffer finalizer owns the allocation - DSL return conversion: defer `deinit` on the local owner, which frees restored/untransferred data and is a no-op after transfer The heap finalizer context is the owning value itself, matching the boxed-owner model rather than duplicating ownership fields in a second context type. Node implementation reference for the ambiguous failure boundary: - https://github.com/nodejs/node/blob/main/src/js_native_api_v8.cc#L3126-L3141 - https://github.com/nodejs/node/blob/main/src/node_api.cc#L1077-L1113
1 parent 3ab8c11 commit b92c2de

8 files changed

Lines changed: 355 additions & 50 deletions

File tree

‎README.md‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,7 @@ Code paths that cannot accept or return DSL classes pass
179179
| `Function` | `Function` | `call(args)` |
180180
| `Value` | `any` | `isNumber()`, `asNumber()`, type checking/narrowing |
181181
| `Uint8Array` etc. | `TypedArray` | `toSlice()`, `toArray(len)`, `from(slice)` |
182+
| `OwnedUint8Array` etc. | `TypedArray` | `fromOwnedSlice(allocator, data)`, `fromSlice(allocator, data)` |
182183
| `Promise(T)` | `Promise` | `resolve(value)`, `reject(err)` |
183184

184185
---
@@ -376,6 +377,24 @@ pub fn sum(data: Uint8Array) !Number {
376377
}
377378
```
378379

380+
Use an owned return type to transfer an allocator-owned slice to JavaScript
381+
without copying its elements:
382+
383+
```zig
384+
pub fn serialize() !js.OwnedUint8Array {
385+
const allocator = js.allocator();
386+
const data = try allocator.alloc(u8, 32);
387+
// Fill data...
388+
return js.OwnedUint8Array.fromOwnedSlice(allocator, data);
389+
}
390+
```
391+
392+
Returning the value transfers its allocation to JavaScript without copying and
393+
leaves the Zig owner empty. Failures before N-API accepts the external memory
394+
leave ownership in Zig so it can be released normally. The allocator must remain
395+
valid until the ArrayBuffer finalizer runs. If external ArrayBuffers are
396+
unsupported, the original error is returned; no copy fallback is performed.
397+
379398
### Promises
380399

381400
```zig
@@ -543,11 +562,11 @@ const callback = napi.createCallback(0, makeExternalBuffer, .{
543562
});
544563
```
545564

546-
`OwnedBuffer.intoValue` consumes the buffer even if conversion fails, so the caller must not
547-
deinitialize it afterwards. Once N-API accepts the external buffer, the allocator must remain valid
548-
until its finalizer runs, including if N-API subsequently reports an error. If the environment
549-
disallows external buffers, `intoValue` copies the bytes and releases the original allocation before
550-
returning.
565+
`OwnedBuffer.intoValue` empties the source after ownership transfers, so a deferred `deinit` is safe.
566+
Failures before N-API accepts the external memory leave ownership in the source; later failures leave
567+
it empty because the finalizer may already own the allocation. Once N-API accepts the external buffer,
568+
the allocator must remain valid until its finalizer runs. If the environment disallows external
569+
buffers, `intoValue` copies the bytes and consumes the source only after that copy succeeds.
551570

552571
### Creating Classes
553572

‎examples/js_dsl/mod.test.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -255,6 +255,14 @@ describe("typed arrays", () => {
255255
expect(result).toBeInstanceOf(Uint8Array);
256256
}
257257
});
258+
259+
it("ownedUint8Array returns allocator-owned native data", () => {
260+
for (const input of [[], [10, 20, 30, 40]]) {
261+
const result = mod.ownedUint8Array(input);
262+
expect(result).toBeInstanceOf(Uint8Array);
263+
expect(Array.from(result)).toEqual(input);
264+
}
265+
});
258266
});
259267

260268
// Section 7: Promises

‎examples/js_dsl/mod.zig‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ const Object = js.Object;
1212
const Function = js.Function;
1313
const Value = js.Value;
1414
const Uint8Array = js.Uint8Array;
15+
const OwnedUint8Array = js.OwnedUint8Array;
1516
const Float64Array = js.Float64Array;
1617
const Promise = js.Promise;
1718

@@ -243,6 +244,20 @@ pub fn externalUint8Array(arr: Array) !Uint8Array {
243244
return Uint8Array.fromExternal(tmp);
244245
}
245246

247+
/// Transfer an allocator-owned native allocation to a JavaScript Uint8Array.
248+
pub fn ownedUint8Array(arr: Array) !OwnedUint8Array {
249+
const len = try arr.length();
250+
const alloc = js.allocator();
251+
const data = try alloc.alloc(u8, len);
252+
errdefer alloc.free(data);
253+
254+
var i: u32 = 0;
255+
while (i < len) : (i += 1) {
256+
data[i] = @intCast((try arr.getNumber(i)).assertI32());
257+
}
258+
return OwnedUint8Array.fromOwnedSlice(alloc, data);
259+
}
260+
246261
// ============================================================================
247262
// Section 7: Promises
248263
// ============================================================================

‎src/OwnedBuffer.zig‎

Lines changed: 70 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
//! Allocator-backed bytes whose ownership can be transferred to JavaScript.
2-
//! Like other Zig owning values, an OwnedBuffer must not be copied or deinitialized after transfer.
2+
//! Like other Zig owning values, an OwnedBuffer must not be copied while it owns data.
33

44
const std = @import("std");
55
const c = @import("c.zig").c;
@@ -11,11 +11,6 @@ data: []u8,
1111

1212
const OwnedBuffer = @This();
1313

14-
const FinalizerContext = struct {
15-
allocator: std.mem.Allocator,
16-
data: []u8,
17-
};
18-
1914
/// Takes ownership of `data`, which must have been allocated by `allocator`.
2015
/// The allocator must remain valid until the buffer is deinitialized or finalized by JavaScript.
2116
pub fn fromOwnedSlice(allocator: std.mem.Allocator, data: []u8) OwnedBuffer {
@@ -30,62 +25,75 @@ pub fn fromSlice(allocator: std.mem.Allocator, data: []const u8) !OwnedBuffer {
3025
return .fromOwnedSlice(allocator, try allocator.dupe(u8, data));
3126
}
3227

33-
/// Releases a buffer that has not been transferred to JavaScript.
28+
/// Releases data that has not been transferred to JavaScript. This is also
29+
/// safe after a successful transfer, when `data` is empty.
3430
pub fn deinit(self: *OwnedBuffer) void {
3531
self.allocator.free(self.data);
3632
self.* = undefined;
3733
}
3834

3935
/// Transfers ownership to a JavaScript Buffer.
40-
/// This consumes the buffer even on failure; the caller must not deinitialize it afterwards.
41-
/// On success, JavaScript releases the allocation through the N-API finalizer.
42-
pub fn intoValue(self: OwnedBuffer, env: Env) !Value {
43-
const allocator = self.allocator;
36+
///
37+
/// On success, `self.data` is empty and JavaScript releases the original
38+
/// allocation through the Buffer finalizer. Failures before N-API accepts the
39+
/// external memory leave ownership in `self`; failures after ownership may
40+
/// have transferred leave `self.data` empty.
41+
///
42+
/// Unsupported external buffers return `error.NoExternalBuffersAllowed`
43+
/// without a copy fallback. The caller may deinitialize `self` after this
44+
/// function returns.
45+
pub fn intoValue(self: *OwnedBuffer, env: Env) !Value {
4446
const data = self.data;
4547

4648
if (data.len == 0) {
47-
defer allocator.free(data);
48-
return try env.createBuffer(0, null);
49+
const value = try env.createBuffer(0, null);
50+
self.allocator.free(data);
51+
self.data = &.{};
52+
return value;
4953
}
5054

51-
const context = try createFinalizerContext(self);
55+
const owner = try moveToHeap(self);
5256

53-
return env.createExternalBuffer(data, finalize, context) catch |err| {
54-
if (err == error.NoExternalBuffersAllowed) {
55-
defer release(context);
56-
return try env.createBufferCopy(data, null);
57+
return env.createExternalBuffer(data, finalize, owner) catch |err| {
58+
switch (err) {
59+
error.NoExternalBuffersAllowed,
60+
error.PendingException,
61+
error.CannotRunJS,
62+
=> restoreFromHeap(self, owner),
63+
else => {},
5764
}
58-
// Other failures may occur after the finalizer has taken ownership.
5965
return err;
6066
};
6167
}
6268

63-
fn createFinalizerContext(self: OwnedBuffer) !*FinalizerContext {
64-
const context = self.allocator.create(FinalizerContext) catch |err| {
65-
self.allocator.free(self.data);
66-
return err;
67-
};
68-
context.* = .{
69-
.allocator = self.allocator,
70-
.data = self.data,
71-
};
72-
return context;
69+
fn moveToHeap(self: *OwnedBuffer) !*OwnedBuffer {
70+
const owner = try self.allocator.create(OwnedBuffer);
71+
owner.* = self.*;
72+
self.data = &.{};
73+
return owner;
74+
}
75+
76+
fn restoreFromHeap(self: *OwnedBuffer, owner: *OwnedBuffer) void {
77+
const allocator = owner.allocator;
78+
std.debug.assert(self.data.len == 0);
79+
self.* = owner.*;
80+
allocator.destroy(owner);
7381
}
7482

7583
fn finalize(
7684
_: c.napi_env,
7785
finalize_data: ?*anyopaque,
7886
finalize_hint: ?*anyopaque,
7987
) callconv(.c) void {
80-
const context: *FinalizerContext = @ptrCast(@alignCast(finalize_hint orelse unreachable));
81-
std.debug.assert(finalize_data == @as(?*anyopaque, @ptrCast(context.data.ptr)));
82-
release(context);
88+
const owner: *OwnedBuffer = @ptrCast(@alignCast(finalize_hint orelse unreachable));
89+
std.debug.assert(finalize_data == @as(?*anyopaque, @ptrCast(owner.data.ptr)));
90+
release(owner);
8391
}
8492

85-
fn release(context: *FinalizerContext) void {
86-
const allocator = context.allocator;
87-
allocator.free(context.data);
88-
allocator.destroy(context);
93+
fn release(owner: *OwnedBuffer) void {
94+
const allocator = owner.allocator;
95+
allocator.free(owner.data);
96+
allocator.destroy(owner);
8997
}
9098

9199
test "OwnedBuffer fromSlice owns an independent copy" {
@@ -97,14 +105,34 @@ test "OwnedBuffer fromSlice owns an independent copy" {
97105
try std.testing.expectEqualSlices(u8, &.{ 1, 2, 3 }, buffer.data);
98106
}
99107

100-
test "OwnedBuffer releases data when finalizer context allocation fails" {
101-
const data = try std.testing.allocator.dupe(u8, "external");
102-
108+
test "OwnedBuffer retains data when moving the owner to the heap fails" {
103109
var failing_allocator = std.testing.FailingAllocator.init(std.testing.allocator, .{
104-
.fail_index = 0,
110+
.fail_index = 1,
105111
});
106-
const buffer = OwnedBuffer.fromOwnedSlice(failing_allocator.allocator(), data);
112+
{
113+
var buffer = try OwnedBuffer.fromSlice(
114+
failing_allocator.allocator(),
115+
"external",
116+
);
117+
defer buffer.deinit();
118+
119+
try std.testing.expectError(error.OutOfMemory, moveToHeap(&buffer));
120+
try std.testing.expectEqualSlices(u8, "external", buffer.data);
121+
try std.testing.expectEqual(@as(usize, 0), failing_allocator.deallocations);
122+
}
107123

108-
try std.testing.expectError(error.OutOfMemory, createFinalizerContext(buffer));
109124
try std.testing.expectEqual(@as(usize, 1), failing_allocator.deallocations);
110125
}
126+
127+
test "OwnedBuffer restores ownership from the heap" {
128+
var buffer = try OwnedBuffer.fromSlice(std.testing.allocator, "external");
129+
defer buffer.deinit();
130+
131+
const owner = try moveToHeap(&buffer);
132+
const source_is_empty = buffer.data.len == 0;
133+
134+
restoreFromHeap(&buffer, owner);
135+
136+
try std.testing.expect(source_is_empty);
137+
try std.testing.expectEqualSlices(u8, "external", buffer.data);
138+
}

‎src/js.zig‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ pub const Function = @import("js/function.zig").Function;
2525
pub const Value = @import("js/value.zig").Value;
2626

2727
pub const TypedArray = typed_arrays.TypedArray;
28+
pub const OwnedTypedArray = typed_arrays.OwnedTypedArray;
2829
pub const Int8Array = typed_arrays.Int8Array;
2930
pub const Uint8Array = typed_arrays.Uint8Array;
3031
pub const Uint8ClampedArray = typed_arrays.Uint8ClampedArray;
@@ -36,6 +37,17 @@ pub const Float32Array = typed_arrays.Float32Array;
3637
pub const Float64Array = typed_arrays.Float64Array;
3738
pub const BigInt64Array = typed_arrays.BigInt64Array;
3839
pub const BigUint64Array = typed_arrays.BigUint64Array;
40+
pub const OwnedInt8Array = typed_arrays.OwnedInt8Array;
41+
pub const OwnedUint8Array = typed_arrays.OwnedUint8Array;
42+
pub const OwnedUint8ClampedArray = typed_arrays.OwnedUint8ClampedArray;
43+
pub const OwnedInt16Array = typed_arrays.OwnedInt16Array;
44+
pub const OwnedUint16Array = typed_arrays.OwnedUint16Array;
45+
pub const OwnedInt32Array = typed_arrays.OwnedInt32Array;
46+
pub const OwnedUint32Array = typed_arrays.OwnedUint32Array;
47+
pub const OwnedFloat32Array = typed_arrays.OwnedFloat32Array;
48+
pub const OwnedFloat64Array = typed_arrays.OwnedFloat64Array;
49+
pub const OwnedBigInt64Array = typed_arrays.OwnedBigInt64Array;
50+
pub const OwnedBigUint64Array = typed_arrays.OwnedBigUint64Array;
3951

4052
pub const Promise = @import("js/promise.zig").Promise;
4153
pub const createPromise = @import("js/promise.zig").createPromise;

0 commit comments

Comments
 (0)