Skip to content

Commit 017780e

Browse files
matthargettmatthargett
authored
perf: keep calls out of the tail-call instruction handlers (#77)
With the tail-call dispatch, a handler that contains a call saves and restores a frame record on every instruction it executes, even when the call is on a path that validated code never takes. On `next` all but three of the 615 handlers had such a call: the `instruction_handler_mismatch` and `stack_underflow` panics, the bounds-check panics of value-stack and global indexing, or the conversion of the `Trap::ValueStackOverflow` a push could return into an `ExecError`. The handlers now `become` the mismatch and instruction-pointer panics instead of calling them. Value-stack and global accesses that validation rules out stop through `invariant_violated`, which with `nightly-tail-calls` in release builds is `core::intrinsics::abort` (an inline trap instruction) and otherwise the panic it was before. A push inside a function body no longer returns a `Result`: `enter_locals` reserves the function's whole operand stack or traps before the body runs (#59). Only modules that skip validation can break these invariants, and archives, which do, are already documented as trusted input. The second commit inlines the five memory helpers that were not `#[inline(always)]`, so a load or store through a local address no longer calls `exec_load_local` and similar. 448 of the 615 handlers are now frameless, and `i32.add` is 23 instructions instead of 31. Memory, call and return handlers still keep a frame for their traps and slow paths. Change in cycles per call against `next` (a0ea681), on the efficiency cores of an iPhone 12 (A14), iPhone XS Max (A12) and iPhone SE (A13), median of five interleaved launches per build: | benchmark | A14 | A12 | A13 | |---|---:|---:|---:| | xmrsplayer (1024-frame buffer) | −5.5% | −4.0% | −12.3% | | audio DSP (1000 frames × 512) | −3.8% | −4.3% | −9.7% | | graphql-validation (AS) | −4.6% | −5.5% | −7.7% | | multi-memory twin: one memory | −9.2% | −8.7% | −11.1% | | crc32 (64 KB) | −5.5% | −10.4% | −7.7% | | convolution 256×256 | −3.5% | −5.9% | −6.1% | | sieve (10000) | −14.1% | −8.6% | −13.9% | | bulk_memory (memory.copy/fill) | −6.7% | −8.3% | −7.6% | | matmul relaxed-simd FMA | −6.0% | −8.4% | −9.7% | | GC binary trees (~130K struct.new) | −1.3% | −2.5% | −1.9% | | fib(30) | −8.7% | −6.4% | −7.6% | | tail-call FSM (65536 return_call) | −8.3% | −6.5% | −7.7% | | call_indirect (200K) | −2.9% | −5.9% | −3.9% | | call_ref (200K) | −4.3% | −3.8% | −5.0% | | vtable_poly4 (200K) | −4.3% | −4.1% | −5.3% | | EH parser, exnref (4096 stmts, 25% throw) | −2.8% | −1.4% | −3.2% | | **geomean, cycles** | **−5.8%** | **−6.0%** | **−7.6%** | | geomean, instructions | −7.2% | −7.2% | −7.2% | | geomean, wall time | −5.7% | −6.0% | −7.7% | | rows faster (cycles) | 16/16 | 16/16 | 16/16 | On an M4's efficiency cores these two commits give −8.6% instructions and −5.6% cycles, and the loop dispatch (without `nightly-tail-calls`) −3.5% instructions and −1.1% cycles. Some gory details: * xmrsplayer cycles per call fall from 39.5M to 37.4M, graphql from 32.3M to 30.8M. * branch mispredicts stay flat, within ±5%. * the main one cost is memory-order flushes: 7.7k → 13.2k per xmrsplayer call, roughly 0.3–0.4% of cycles against the 5.5% gain. I elaborated on the memory-order flushes in a separate PR. The good news is that the extra inlining isn't busting the cache, and the call avoidance isn't unraveling some other CPU performance in a way I didn't predict. we'll keep slamming into this memory-order flush stall pileup until we deal with it explicitly, but it's still net positive on wall clock and benchmarks even on the cut-down efficiency cores :)
1 parent a0ea681 commit 017780e

10 files changed

Lines changed: 189 additions & 104 deletions

File tree

‎crates/tinywasm/src/interpreter/executor/dispatch_become.rs‎

Lines changed: 43 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -6,12 +6,6 @@ struct Bounded;
66
type UnbudgetedHandler = for<'store> fn(&mut Executor<'store>, &[Instruction], usize, Instruction) -> ExecResult<()>;
77
type BoundedHandler = for<'store> fn(&mut Executor<'store>, usize, Instruction, u32) -> ExecResult<()>;
88

9-
#[cold]
10-
#[inline(never)]
11-
fn instruction_handler_mismatch() -> ! {
12-
unreachable!("instruction handler mismatch")
13-
}
14-
159
macro_rules! define_unbudgeted_tail_dispatch {
1610
($executor:ident, $instr_ptr:ident, $dispatch_next:ident, $dispatch_flow:ident;
1711
$($variant:ident $(($($arg:pat),*))? $({ $($field:ident),* })? => $body:expr),* $(,)?) => {
@@ -36,9 +30,11 @@ macro_rules! define_unbudgeted_tail_dispatch {
3630
macro_rules! $dispatch_next {
3731
($next_instr_ptr:expr) => {{
3832
let next_instr_ptr = $next_instr_ptr;
39-
let instruction = instructions[next_instr_ptr];
40-
let handler = Self::handler_for(instruction.opcode());
41-
become handler($executor, instructions, next_instr_ptr, instruction);
33+
let Some(&next) = instructions.get(next_instr_ptr) else {
34+
become Self::invalid_instr_ptr($executor, instructions, next_instr_ptr, instruction);
35+
};
36+
let handler = Self::handler_for(next.opcode());
37+
become handler($executor, instructions, next_instr_ptr, next);
4238
}};
4339
}
4440
macro_rules! $dispatch_flow {
@@ -55,10 +51,10 @@ macro_rules! define_unbudgeted_tail_dispatch {
5551
}
5652
use tinywasm_types::Instruction::*;
5753
$(let $variant($($arg),*) = &instruction else {
58-
cold!(instruction_handler_mismatch())
54+
become Self::handler_mismatch($executor, instructions, $instr_ptr, instruction);
5955
};)?
6056
$(let $variant { $($field),* } = &instruction else {
61-
cold!(instruction_handler_mismatch())
57+
become Self::handler_mismatch($executor, instructions, $instr_ptr, instruction);
6258
};)?
6359
$body;
6460
$dispatch_next!($instr_ptr + 1)
@@ -98,9 +94,11 @@ macro_rules! define_bounded_tail_dispatch {
9894
});
9995
}
10096

101-
let instruction = $executor.func.instructions[next_instr_ptr];
102-
let handler = Self::handler_for(instruction.opcode());
103-
become handler($executor, next_instr_ptr, instruction, instructions_until_checkpoint - 1);
97+
let Some(&next) = $executor.func.instructions.get(next_instr_ptr) else {
98+
become Self::invalid_instr_ptr($executor, next_instr_ptr, instruction, instructions_until_checkpoint);
99+
};
100+
let handler = Self::handler_for(next.opcode());
101+
become handler($executor, next_instr_ptr, next, instructions_until_checkpoint - 1);
104102
}};
105103
}
106104
macro_rules! $dispatch_flow {
@@ -116,10 +114,10 @@ macro_rules! define_bounded_tail_dispatch {
116114
}
117115
use tinywasm_types::Instruction::*;
118116
$(let $variant($($arg),*) = &instruction else {
119-
cold!(instruction_handler_mismatch())
117+
become Self::handler_mismatch($executor, $instr_ptr, instruction, instructions_until_checkpoint);
120118
};)?
121119
$(let $variant { $($field),* } = &instruction else {
122-
cold!(instruction_handler_mismatch())
120+
become Self::handler_mismatch($executor, $instr_ptr, instruction, instructions_until_checkpoint);
123121
};)?
124122
$body;
125123
$dispatch_next!($instr_ptr + 1)
@@ -130,11 +128,40 @@ macro_rules! define_bounded_tail_dispatch {
130128

131129
impl Unbudgeted {
132130
instruction_handlers!(define_unbudgeted_tail_dispatch);
131+
132+
// The handlers tail-call these cold paths instead of calling them: a call would make every
133+
// handler save a stack frame.
134+
135+
#[cold]
136+
#[inline(never)]
137+
fn handler_mismatch(_: &mut Executor<'_>, _: &[Instruction], _: usize, _: Instruction) -> ExecResult<()> {
138+
unreachable!("instruction handler mismatch")
139+
}
140+
141+
#[cold]
142+
#[inline(never)]
143+
fn invalid_instr_ptr(_: &mut Executor<'_>, _: &[Instruction], instr_ptr: usize, _: Instruction) -> ExecResult<()> {
144+
unreachable!("instruction pointer {instr_ptr} out of range, this is a bug")
145+
}
133146
}
134147

135148
impl Bounded {
136149
instruction_handlers!(define_bounded_tail_dispatch);
137150

151+
// Tail-called like `Unbudgeted`'s.
152+
153+
#[cold]
154+
#[inline(never)]
155+
fn handler_mismatch(_: &mut Executor<'_>, _: usize, _: Instruction, _: u32) -> ExecResult<()> {
156+
unreachable!("instruction handler mismatch")
157+
}
158+
159+
#[cold]
160+
#[inline(never)]
161+
fn invalid_instr_ptr(_: &mut Executor<'_>, instr_ptr: usize, _: Instruction, _: u32) -> ExecResult<()> {
162+
unreachable!("instruction pointer {instr_ptr} out of range, this is a bug")
163+
}
164+
138165
#[inline(always)]
139166
fn run(executor: &mut Executor<'_>) -> ExecResult<()> {
140167
let instr_ptr = executor.cf.instr_ptr;

‎crates/tinywasm/src/interpreter/executor/instructions.rs‎

Lines changed: 21 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -3,36 +3,37 @@ macro_rules! exec_op {
33
fn exec_binary_fallible(value_stack: &mut ValueStack) -> Result<(), Trap> {
44
let $rhs = <$ty>::stack_pop(value_stack);
55
let $lhs = <$ty>::stack_pop(value_stack);
6-
<$ty>::stack_push(value_stack, $expr?)
6+
<$ty>::stack_push(value_stack, $expr?);
7+
Ok(())
78
}
89
exec_binary_fallible(&mut $executor.store.value_stack)?;
910
}};
1011
($executor:ident; unary $from:ty => $to:ty, |$v:ident| $expr:expr) => {{
11-
fn exec_unary(value_stack: &mut ValueStack) -> Result<(), Trap> {
12+
fn exec_unary(value_stack: &mut ValueStack) {
1213
let $v = <$from>::stack_pop(value_stack);
13-
<$to>::stack_push(value_stack, $expr)
14+
<$to>::stack_push(value_stack, $expr);
1415
}
15-
exec_unary(&mut $executor.store.value_stack)?;
16+
exec_unary(&mut $executor.store.value_stack);
1617
}};
1718
($executor:ident; binary $from:ty => $to:ty, |$lhs:ident, $rhs:ident| $expr:expr) => {{
1819
exec_op!($executor; binary $from, $from => $to, |$lhs, $rhs| $expr)
1920
}};
2021
($executor:ident; binary $lhs_ty:ty, $rhs_ty:ty => $res:ty, |$lhs:ident, $rhs:ident| $expr:expr) => {{
21-
fn exec_binary(value_stack: &mut ValueStack) -> Result<(), Trap> {
22+
fn exec_binary(value_stack: &mut ValueStack) {
2223
let $rhs = <$rhs_ty>::stack_pop(value_stack);
2324
let $lhs = <$lhs_ty>::stack_pop(value_stack);
24-
<$res>::stack_push(value_stack, $expr)
25+
<$res>::stack_push(value_stack, $expr);
2526
}
26-
exec_binary(&mut $executor.store.value_stack)?;
27+
exec_binary(&mut $executor.store.value_stack);
2728
}};
2829
($executor:ident; ternary $from:ty => $to:ty, |$a:ident, $b:ident, $c:ident| $expr:expr) => {{
29-
fn exec_ternary(value_stack: &mut ValueStack) -> Result<(), Trap> {
30+
fn exec_ternary(value_stack: &mut ValueStack) {
3031
let $c = <$from>::stack_pop(value_stack);
3132
let $b = <$from>::stack_pop(value_stack);
3233
let $a = <$from>::stack_pop(value_stack);
33-
<$to>::stack_push(value_stack, $expr)
34+
<$to>::stack_push(value_stack, $expr);
3435
}
35-
exec_ternary(&mut $executor.store.value_stack)?;
36+
exec_ternary(&mut $executor.store.value_stack);
3637
}};
3738
}
3839

@@ -99,9 +100,9 @@ macro_rules! instruction_handlers {
99100
Return32 => dispatch_flow!(executor.exec_return_32()),
100101
Return64 => dispatch_flow!(executor.exec_return_64()),
101102
Return128 => dispatch_flow!(executor.exec_return_128()),
102-
LocalGet32(local_index) => Value32::local_push(&mut executor.store.value_stack, &executor.cf, *local_index)?,
103-
LocalGet64(local_index) => Value64::local_push(&mut executor.store.value_stack, &executor.cf, *local_index)?,
104-
LocalGet128(local_index) => Value128::local_push(&mut executor.store.value_stack, &executor.cf, *local_index)?,
103+
LocalGet32(local_index) => Value32::local_push(&mut executor.store.value_stack, &executor.cf, *local_index),
104+
LocalGet64(local_index) => Value64::local_push(&mut executor.store.value_stack, &executor.cf, *local_index),
105+
LocalGet128(local_index) => Value128::local_push(&mut executor.store.value_stack, &executor.cf, *local_index),
105106
LocalSet32(local_index) => executor.exec_local_set_pop::<Value32>(*local_index),
106107
LocalSet64(local_index) => executor.exec_local_set_pop::<Value64>(*local_index),
107108
LocalSet128(local_index) => executor.exec_local_set_pop::<Value128>(*local_index),
@@ -201,10 +202,10 @@ macro_rules! instruction_handlers {
201202
GlobalTee32(global_index) => executor.exec_global_tee::<Value32>(*global_index),
202203
GlobalTee64(global_index) => executor.exec_global_tee::<Value64>(*global_index),
203204
GlobalTee128(global_index) => executor.exec_global_tee::<Value128>(*global_index),
204-
Const32(val) => i32::stack_push(&mut executor.store.value_stack, *val)?,
205-
Const64Imm(val) => i64::stack_push(&mut executor.store.value_stack, i64::from(*val))?,
206-
Const64(idx) => i64::stack_push(&mut executor.store.value_stack, idx.resolve(&executor.func.data).value())?,
207-
Const128Imm(val) => Value128::stack_push(&mut executor.store.value_stack, Value128(u128::from(*val).to_le_bytes()))?,
205+
Const32(val) => i32::stack_push(&mut executor.store.value_stack, *val),
206+
Const64Imm(val) => i64::stack_push(&mut executor.store.value_stack, i64::from(*val)),
207+
Const64(idx) => i64::stack_push(&mut executor.store.value_stack, idx.resolve(&executor.func.data).value()),
208+
Const128Imm(val) => Value128::stack_push(&mut executor.store.value_stack, Value128(u128::from(*val).to_le_bytes())),
208209
I64Eqz => exec_op!(executor; unary i64 => i32, |v| i32::from(v == 0)),
209210
I32Eqz => exec_op!(executor; unary i32 => i32, |v| i32::from(v == 0)),
210211
I32Eq => exec_op!(executor; binary i32 => i32, |a, b| i32::from(a == b)),
@@ -289,8 +290,8 @@ macro_rules! instruction_handlers {
289290
I64Popcnt => exec_op!(executor; unary i64 => i64, |v| i64::from(v.count_ones())),
290291

291292
// Reference types
292-
RefFunc(func_idx) => ValueRef::stack_push(&mut executor.store.value_stack, ValueRef::from_category_addr(executor.module.resolve_func_addr(*func_idx)))?,
293-
RefNull(_) => ValueRef::stack_push(&mut executor.store.value_stack, ValueRef::NULL)?,
293+
RefFunc(func_idx) => ValueRef::stack_push(&mut executor.store.value_stack, ValueRef::from_category_addr(executor.module.resolve_func_addr(*func_idx))),
294+
RefNull(_) => ValueRef::stack_push(&mut executor.store.value_stack, ValueRef::NULL),
294295
RefIsNull => executor.exec_ref_is_null()?,
295296
RefAsNonNull => executor.exec_ref_as_non_null()?,
296297
RefI31 => exec_op!(executor; unary i32 => ValueRef, |v| ValueRef::from_i31(v)),
@@ -459,7 +460,7 @@ macro_rules! instruction_handlers {
459460
V128Store64Lane(arg) => executor.exec_mem_store_lane::<i64, 8>(*arg)?,
460461
V128Load32Zero(idx) => executor.exec_mem_load::<i32, 4, Value128>(idx.resolve(&executor.func.data), |v| Value128::from_i32x4([v, 0, 0, 0]))?,
461462
V128Load64Zero(idx) => executor.exec_mem_load::<i64, 8, Value128>(idx.resolve(&executor.func.data), |v| Value128::from_i64x2([v, 0]))?,
462-
Const128(arg) => Value128::stack_push(&mut executor.store.value_stack, Value128(arg.resolve(&executor.func.data).value()))?,
463+
Const128(arg) => Value128::stack_push(&mut executor.store.value_stack, Value128(arg.resolve(&executor.func.data).value())),
463464
I8x16ExtractLaneS(lane) => executor.exec_simd_extract_lane::<i32>(*lane, |v, lane| v.extract_lane_i8(lane) as i32)?,
464465
I8x16ExtractLaneU(lane) => executor.exec_simd_extract_lane::<i32>(*lane, |v, lane| v.extract_lane_u8(lane) as i32)?,
465466
I16x8ExtractLaneS(lane) => executor.exec_simd_extract_lane::<i32>(*lane, |v, lane| v.extract_lane_i16(lane) as i32)?,

0 commit comments

Comments
 (0)