The problem in one sentence: when a server hands work to an accelerator that takes microseconds to a few milliseconds, the CPU has nothing good to do with the wait. Blocking costs a context switch comparable to the offload itself. Busy-waiting burns the core. The third option is to overlap — run another request's work during the wait — and that normally means rewriting the server as async code.
This runtime does the overlap without the rewrite.
libtransparent.so is an LD_PRELOAD M:N fiber runtime. It interposes a small
set of standard symbols and nothing application-specific:
| Interposed | Becomes |
|---|---|
pthread_create |
spawn a fiber on a carrier thread, not an OS thread |
read, write, recv, send, poll |
non-blocking + yield the fiber on EAGAIN |
accel_run / accel_encrypt |
accel_submit + yield; resume on completion |
pthread_mutex_lock |
fiber-aware: park and retry rather than deadlock the carrier |
pthread_cond_*, pthread_rwlock_* |
fiber-aware equivalents |
accept, accept4 |
mark the process as serving (see below) |
A single carrier thread runs a scheduler loop. Each iteration it admits new
fibers, polls epoll for I/O readiness, polls the accelerator for completions,
drains cross-thread wakeups, checks timed waits, and then runs every runnable
fiber until it yields.
application (unmodified) libtransparent.so
------------------------ ---------------------------------
accept() ───► mark serving
pthread_create(handler) ───► make fiber, queue it
handler: carrier loop:
read(fd, ...) ───► EAGAIN? park fiber on epoll ─┐
run the next runnable fiber │
accel_run(buf, n) ───► accel_submit + park ─────────┤
run the next runnable fiber │
accel_done? wake it ◄────────┤
write(fd, ...) ───► ... and so on ◄──────────────┘
The application never learns any of this happened. It is still plain blocking code; it just stopped being the only thing on its core.
The context switch is the whole point. fw_switch.S saves callee-saved
registers and the stack pointer, and nothing else — no sigprocmask syscall,
unlike swapcontext(3). A switch is tens of nanoseconds, which is what makes
it worth switching away for a 20 µs offload at all. This is a clean-room
reproduction of the FastWake fast context switch (APNet 2023). x86-64 and
aarch64 are both supported.
A server creates threads at startup for reasons that have nothing to do with serving connections — log writers, timers, worker pools. Turning those into fibers would collapse them onto one core and change the program's behavior.
So pthread_create only makes fibers after the first successful accept().
Before that, threads are real threads. It is a heuristic, and it is the right
one for thread-per-connection servers, which create their handler thread
immediately after accepting. TOFFLOAD_POOL=1 overrides it for thread-pool
servers where the workers are created up front.
Turning threads into fibers changes when handlers can interleave. Two things break if you ignore that.
Thread-local state. All fibers on a carrier share one OS thread, so they
share its errno and OpenSSL error queue. Handler A sets errno, yields,
handler B overwrites it, A resumes and reads B's value. Measured: 100%
corruption without mitigation. The runtime saves and restores per-fiber errno
and the OpenSSL ERR queue at every yield.
Shared state across connections. A read-modify-write that spans the offload is atomic between OS threads (the thread just blocks) but not between fibers (the fiber yields, and a peer runs in the gap). Unlocked shared state that was fine before can lose updates. Three answers, in order of preference:
- If the application locks the state, the fiber-aware mutex respects it. A naive interposer would deadlock the carrier here; the fiber-aware one parks the waiter.
- The conflict detector finds the hazard with no application cooperation: write-protect the writable data segments, snapshot a version clock when a fiber parks at the offload, and treat a post-offload write fault to a page that changed during the park as a conflict.
TOFFLOAD_ENFORCE=1serializes conflicting handlers with a handler lock held from request read to response write.
safety.md covers this properly, including what the detector costs and where it is imprecise.
Four functions, described in include/toffload/accel.h:
long accel_submit(unsigned char *buf, int n); /* start work, return a handle, do not block */
int accel_done(long id); /* cheap non-blocking poll */
void accel_release(long id); /* free the handle */
void accel_run(unsigned char *buf, int n); /* blocking; what the app calls */The runtime only ever calls the first three. accel_run is the application's
entry point and the thing the runtime replaces. Three backends ship: an
emulated device (real AES-CTR plus a latency knob), CUDA (AES on the GPU, one
stream per request, cudaEventQuery as the completion poll), and a remote
RSA-2048 signer over TCP. See writing-a-backend.md.
The runtime gives overlap to thread-per-connection servers. Event-loop
servers (nginx, redis, memcached) have no per-connection thread to fiberize;
they run safely under the runtime but gain nothing, because getting overlap
there would mean the runtime owning the event loop. Servers that block below
libc — InnoDB's raw futex and io_uring_enter — cannot be yielded through at
all. is-my-server-supported.md turns this into a
five-minute test you can run against your own binary.
| Path | What lives there |
|---|---|
src/libtransparent.c |
the interposers and the carrier scheduler |
src/fw_fiber.c, src/fw_switch.S |
fiber creation and the context switch |
src/detector.c |
page-protection conflict detector |
src/config.c |
knob resolution: env, legacy env, config file |
include/toffload/ |
the public headers |
backends/ |
emulated, cuda, remote, tls |
examples/servers/ |
the servers the tests and demo drive |
examples/integrations/ |
ten real servers with minimal-edit offload |