Skip to content

configuration.md

Latest commit

 

History

History
141 lines (108 loc) · 5.54 KB

configuration.md

File metadata and controls

141 lines (108 loc) · 5.54 KB
Raw

Configuration reference

Every knob is read once, when the runtime loads. Three sources, first hit wins:

  1. environment TOFFLOAD_<KNOB>
  2. environment RT_<KNOB> — the legacy prefix, still supported so the paper's command lines keep working
  3. the file named by TOFFLOAD_CONFIG

Environment beats file deliberately: you can override a deployed config for one run without editing it.

Config file format

# /etc/toffload.conf — '#' starts a comment anywhere on a line
core     = 4
enforce  = yes
stats    = /var/run/toffload.stats:5
no-fiber = 0
TOFFLOAD_CONFIG=/etc/toffload.conf LD_PRELOAD=/usr/local/lib/libtransparent.so ./myserver

Keys are matched case-insensitively with - and _ ignored, so no-fiber, no_fiber and NOFIBER all name the same knob. Values may be quoted to keep surrounding spaces. Booleans are false for 0, false, off, no and the empty string, true for anything else. A missing or malformed file is reported on stderr and is not fatal — the runtime falls back to environment and defaults.

Knobs

Everyday

Knob Default Meaning
STATS off stderr, or a file path, optionally :<seconds> for the period. See below.
QUIET off Suppress the one-line banner printed when the runtime loads.
CORE 4 CPU the carrier pins to, when RT is also set. Ignored otherwise.
POOL off Also fiberize threads created before the first accept(). Needed for thread-pool servers whose workers are created at startup.
ENFORCE off Serialize conflicting handlers with a handler lock held from request read to response write. Correctness over throughput; see safety.md.

Tuning and benchmarking

Knob Default Meaning
RT off Pin the carrier to CORE and run it SCHED_FIFO. Reproduces the paper's microbenchmark setup. Needs privileges, and it can starve a real OS thread a fiber is waiting on — do not use it in production.

Diagnostics: turning individual interposers off

Each of these disables one piece of the runtime. They exist for bisecting a misbehaving application: turn things off until it works, and the last thing you turned off is where the problem is.

Knob Disables
NOFIBER fiberization entirely — pthread_create makes real threads again
NOIO yield-on-EAGAIN for read/write/recv/send
NOMUTEX the fiber-aware mutex (falls back to the real one)
NOCOND the fiber-aware condition variables
NORW the fiber-aware reader-writer locks
NOTLS per-fiber thread-local shadowing
NOSELF pthread_self returning a per-fiber identity
NODETACH pthread_detach interposition
PASSIVE all of the above except NOFIBER — fibers still run, everything else passes through
REALMUTEX uses the naive mutex, which deadlocks the carrier. Present to demonstrate why the fiber-aware one is needed; not for real use.
FIBERSELF force per-fiber pthread_self even where it is otherwise skipped

A good bisection order: PASSIVE first (does the problem survive with only fiberization on?), then NOFIBER (does it go away entirely?). If NOFIBER fixes it, the problem is in fiberization; if PASSIVE fixes it, the problem is in one of the synchronization interposers, and you can narrow it down from there.

Stats

Off by default. This is a library loaded into someone else's process, and periodic chatter on their stderr is a bug, not a feature.

TOFFLOAD_STATS=stderr        # every 2 seconds to stderr
TOFFLOAD_STATS=stderr:0.5    # every half second
TOFFLOAD_STATS=/run/tof.txt  # rewrite this file every 2 seconds
TOFFLOAD_STATS=/run/tof.txt:10

One line, stable field order:

fibers_created=64 fibers_live=64 switches=2817174 offloads=708339 \
io_yields=1416678 det_faults=0 det_conflicts=0 serialized=0
Field Meaning
fibers_created connection handlers turned into fibers since start
fibers_live currently allocated, running or parked
switches fiber context switches
offloads offload calls decomposed into submit + yield
io_yields fiber parks at socket I/O
det_faults conflict-detector page faults taken
det_conflicts cross-fiber conflicts found
serialized handlers that had to wait for the handler lock

fibers_created is the field to look at first. If it is 0, the runtime loaded but never engaged — safe, but doing nothing for you. See is-my-server-supported.md.

A final snapshot is emitted at exit whenever stats are enabled.

For programmatic access, link against <toffload/toffload.h> and call toffload_get_stats(). Every symbol there is weak, so the same binary runs with or without the runtime.

Accelerator backend knobs

These belong to the backends, not the runtime, and keep the ACCEL_ prefix.

Emulated (backends/emulated)

Variable Default Meaning
ACCEL_LAT_US 20 Modelled device latency, microseconds. The main dial in the demo.
ACCEL_OP aes aes for AES-128-CTR, zip for a real deflate+inflate round trip.
ACCEL_CORE 6 CPU the emulated device thread pins to.

CUDA (backends/cuda)

Variable Default Meaning
ACCEL_AES_BYTES 1048576 Bytes per offload. 1 MiB models bulk TLS; set 4096 for the launch-bound regime.

Remote (backends/remote)

Variable Default Meaning
ACCEL_REMOTE_HOST 127.0.0.1 Signer address.
ACCEL_REMOTE_PORT 7900 Signer port.