Skip to content

README.md

Latest commit

 

History

History
243 lines (180 loc) · 17.7 KB

README.md

File metadata and controls

243 lines (180 loc) · 17.7 KB
Raw

TYPHOON Evaluation

A Docker-based harness that captures TYPHOON traffic alongside other protocols and common UDP traffic classes, and answers the central question: can a passive observer distinguish TYPHOON's wire-level characteristics — packet sizes, inter-arrival times, byte entropy, handshake structure — from the rest of the mix?

For the protocol itself, see PROTOCOL.md.

Contents

evaluation/
├── protocols/             # Dockerfiles for the 16 comparison protocols
├── transport/             # shared Rust crate: the senders/sinks every protocol runs
├── background/            # Dockerfiles for the 8 UDP-traffic generators + open-set unknown
├── compose/, observer/    # docker-compose stacks and the tcpdump observer container
├── chaos/                 # tc/netem sidecar for latency / jitter / loss
├── src/typhoon_eval/      # Python orchestrator, parsers, plots, ML
│   ├── shared/            # capture, parse pcaps, common stats
│   ├── self/              # Part 1
│   ├── protocols_op/      # Part 2
│   └── background/        # Part 3 corpus + ML (features + classifiers shared; ml_blending = Test C, detectability/ = Tests A/B/D/E/F)
└── results/               # generated pcaps, plots, ML artefacts (gitignored)

Experiments

Four independent parts; each answers one question.

Part 1 — Self-comparison

Is TYPHOON's traffic profile stable and reproducible? Runs TYPHOON N times under identical or varying conditions and overlays size + IAT distributions. Tight envelopes under identical config = stable; clearly separated envelopes across scenarios = TYPHOON adapts to workload.

Part 2 — Operational comparison

How does TYPHOON compare on reliability and cost to 15 other UDP/TCP secure-transport protocols? Runs a per-packet latency ping against each: a spaced sequence of small equal-sized probes (500 × 256 B @ 20 ms) that the sink echoes, measuring per-packet round-trip time in tunnel-like conditions. It reports delivery (what fraction the receiver gets / echoes back) and the RTT distribution. See Operational metrics for what actually discriminates the protocols (spoiler: the tail, not the median). No classifiers — a closed-world classifier across these 16 protocols would score near 100 % because each protocol has a distinct wire footprint by design, which says nothing about Part 3's question.

Protocols compared: raw_udp, raw_tcp, tls, wireguard, quic, obfs4 (×3 IAT modes), amneziawg, hysteria2, shadowsocks, tor, vless_reality, openvpn, wireguard_daita, typhoon.

Part 3 — Background-blending

Can a passive observer pick TYPHOON out of a realistic UDP traffic mix? Every corpus run captures all 8 TYPHOON mimicry profiles alongside all 8 generators producing common UDP traffic classes (QUIC d/l + u/l, DNS, RTP voice/video, gaming, WireGuard idle, control plane) plus one open-set unknown class held out from training — no per-run sampling, so no profile/class is ever absent from a run its peers are in. Every profile/class contributes exactly one flow per run, except raw_default/tuned_default: they exercise the protocol's genuine auto-fill flow selection and so may contribute 1–3 flows per run (see background/corpus.py's module docstring). Six ML setups, each modelling a different threat model. Test C is the primary blending metric and lives in ml_blending.py; Tests A, B, D, E and F live in the detectability/ package (pair_binary.py = A, closed_world.py = B, open_set.py = D/E/F). Both share the feature pipeline in features.py:

Test Threat model TYPHOON wins when…
A Pair-binary Observer suspects TYPHOON-as-X and trains a binary classifier on real X vs TYPHOON-as-X AUC near 0.5
B Closed-world Observer has labels for every class incl. TYPHOON TYPHOON recall low (often confused with a real class)
C Open-world threshold Observer has labels for background only and flags low-confidence flows; every Barradas classifier (rf/dt/xgb) is evaluated and the adversary-strongest one headlines TPR @ 1 % FPR low
D Open-set binary Observer has labels for TYPHOON + a subset of background classes; unseen classes + unknown held out at test time high FPR on unseen background and unknown
E One-class TYPHOON Observer has only TYPHOON labels (e.g. from a leaked client) trained into a one-class SVM, evaluated against pooled background high FPR on unknown
F One-class + partial catalogue Same one-class TYPHOON SVM as E, but the FPR breakdown reuses D's 3-of-7 background hold-out — models a leaked-client observer who also has a partial protocol catalogue used post-hoc as a filter; bridges D and E high FPR on unseen background and unknown

NB! Tests A/B/D/E/F cross-validate with GroupKFold, grouped by corpus run id, instead of Barradas USENIX'18's plain non-grouped KFold — a run's flows share one chaos (latency/jitter/loss) draw, so an ungrouped split could train and test on flows from the same run. Tests D/E/F additionally restrict every evaluation bucket (held-out background, unseen classes, unknown, per-class breakdown) to the fold's test-run set, so a background flow from a run that fed training is never scored as if it were independently held out. See background/detectability/cli.py's module docstring and each test's own docstring for the full rationale.

Part 4 — Rust-level benchmarking

How fast is the TYPHOON implementation itself? cargo bench roundtrip/handshake timings plus perf-based flamegraphs (kept as interactive .svg and a static .pdf for embedding) for every example binary. Linux only (needs perf + cargo-flamegraph on the host) — mirrors .github/workflows/evaluation.yaml and auto-skips on non-Linux hosts.

Requirements

  • Docker (or rootful Podman — rootless cannot grant NET_ADMIN to the observer).
  • Python 3.11+ with Poetry. poetry install pulls in everything needed for Parts 1–3, including scikit-learn and XGBoost.
  • Optional, Linux only: perf + cargo-flamegraph — enables Part 4 (benchmark). Without them, poe benchmark / the pipeline's benchmark phase is skipped. Flamegraphs additionally need perf sampling access: set kernel.perf_event_paranoid low enough (see Part 4) — otherwise cargo flamegraph fails and the flamegraph step is skipped (the cargo bench numbers are still produced). rsvg-convert (librsvg2-bin) is also recommended to additionally render flamegraphs as .pdf; without it, only the interactive .svg is produced.

Installation

cd evaluation
poetry install                # Python deps (incl. scikit-learn, XGBoost)
poetry poe build               # build the 15 comparison-protocol images (once)
poetry poe background-build    # build the 9 background generator images (once, only for Part 3)

Running the experiments

All commands run from evaluation/ and accept --help for the full flag list. The most common invocations:

Part 1 — Self-comparison (CLI)

poe self-compare              # repeat TYPHOON N times under identical config
poe traffic-compare           # 4 payload × wait scenarios
poe use-case-compare          # one capture per PROTOCOL.md use case
poe plot --example heavy_traffic --out-dir out/   # per-flow packet-structure SVG

Part 2 — Operational comparison (CLI)

poe capture --all             # per-packet latency ping across all 16 protocols
poe capture --all --chaos     # …under netem (2% loss, 100 ms±30 ms delay)
poe analyze                   # parse pcaps → stats.json (detectability metrics)
poe proto-compare             # plots + comparison table for the latest run

Useful capture flags: --protocol <name> (single protocol); --chaos + --loss-pct <n>; --seed <n> (reproducible). --profile <name> selects the TYPHOON settings profile (fake-body mode, decoys); bulk_upload is the operational default, the rest are the Part-3 mimicry profiles. analyze/proto-compare take --run YYYYMMDD_HHMMSS to target an earlier run.

Part 3 — Background-blending (CLI)

poe background-corpus         # every TYPHOON profile + bg class per run (default 70 runs)
poe background-blending       # Test C — confident-blend fraction (primary metric)
poe background-detectability  # Tests A, B, D, E, F held-out detectability scores
poe background-distplot       # per-pair size/IAT distribution overlays

Part 4 — Rust-level benchmarking (CLI)

poe benchmark                 # cargo bench (roundtrip, handshake) + example flamegraphs

Linux only — requires perf and cargo-flamegraph already installed on the host (see .github/workflows/evaluation.yaml for the one-time setup commands); this task does not install them for you. Each flamegraph is kept as the interactive .svg (search/zoom) plus a .pdf rendered via rsvg-convert (librsvg2-bin) for embedding in reports; without rsvg-convert, only the .svg is produced.

perf sampling access. cargo flamegraph runs perf record, which needs kernel.perf_event_paranoid low enough to sample CPU events. To enable it:

echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid      # this session
echo 'kernel.perf_event_paranoid = -1' | sudo tee /etc/sysctl.d/99-perf.conf  # persist across reboots

Alternatively, run the benchmark under sudo (perf as root ignores the paranoid setting), though that also builds/runs cargo as root.

Pipeline

poe evaluate                    # build → capture → analyze → Part 1 + 2 plots → background → benchmark → report.md
poe evaluate --skip background  # everything except the 7500-run Part 3 corpus
poe evaluate --skip build       # reuse existing Docker images
poe evaluate --skip benchmark   # skip cargo bench + flamegraphs (auto-skipped on non-Linux anyway)
poe evaluate --quiet-build      # build phase output → logs/build.log only (docker compose build is 1000s of lines)

# Re-analyze already-stored PCAPs without regenerating the corpus — e.g. after
# changing feature sets or classifier options:
poe evaluate --skip build,capture --corpus-root results/background/pipeline_<id>

# Validation-scale run — every phase, tiny sample sizes (~minutes, not days).
# This is what CI runs on every push; see .github/workflows/evaluation.yaml.
poe evaluate --classification-runs 1 --typhoon-runs 2 --typhoon-uc-runs 1 \
    --typhoon-traffic-runs 1 --background-runs 15 --quiet-build

poe clean                       # delete results/captures and results/background

Reading the results

Outputs are split into two trees:

  • results/ — raw PCAPs (captures and background corpora). Too large to ship; regenerable via poe evaluate.
  • artifacts/<pipeline_id>/ — every derived output (plots, tables, stats.json, top-level report.md). Designed to be zipped and uploaded as a conference artifact bundle.
results/
├── captures/run_<timestamp>/
│   ├── <protocol>.pcap            # raw capture (handshake + data)
│   ├── stats.json                 # per-pcap detectability metrics — see below
│   ├── metadata.json              # delivery %, per-packet RTT distribution (rtt_*)
│   └── logs/<protocol>/           # client + server + observer container logs
└── background/pipeline_<id>/run_*/  # Part 3 per-run pcaps + metadata

artifacts/pipeline_<timestamp>/
├── pipeline_config.json           # resolved CLI parameters
├── report.md                      # top-level index with links to everything below
├── logs/<phase>.log               # per-phase invocation logs
├── analyze/run_*/stats.json       # copies of per-run stats
├── proto_compare/                 # Part 2 PDFs + markdown comparison table
├── flow_plots/                    # Part 1 per-flow packet-structure PDFs
├── self_compare/, use_case_compare/, traffic_compare/  # Part 1 PDFs + JSON
└── background/                    # Part 3 derived outputs (no PCAPs)
    ├── corpus_metadata/run_*/{metadata,config}.json
    ├── blending/blending.json     # per-classifier blend metrics + headline (adversary-strongest) + per-profile
    ├── detectability/             # Tests A, B, D, E, F PDFs + JSON
    └── distplot/                  # per-pair size/IAT overlays PDFs + JSON

Operational metrics (delivery / latency)

Part 2's numbers come from the two endpoints (recorded in metadata.json), not the pcap:

Metric Meaning
delivery_pct probes the sink received ÷ sent (one-way); roundtrip_delivery_pct is the client's echoes-received ÷ sent
rtt_min / p50 / p95 / p99_ms, rtt_jitter_ms per-packet round-trip distribution

Read the tail, not the median. The p50 RTT is dominated by the shared path — the observer hop in clean mode, the netem delay under chaos — and is ~identical across protocols, so it is not a discriminator. The signal lives in two places:

  • Delivery separates best-effort from reliable: under chaos, UDP-family transports settle at the raw ~2 % one-way loss (they drop it) while TCP/QUIC/proxies stay 100 % (they retransmit).
  • The latency tail (p95/p99) exposes the cost of a protocol's design that the median hides:
    • clean — traffic-shapers pay a visible tail (wireguard_daita p95 ~45 ms, obfs4 IAT-mode ~24 ms) from their padding / inter-arrival machines, while plain transports sit at ~7 ms;
    • chaos — reliable transports pay a retransmit spike (p99 ~600–850 ms) that best-effort UDP does not (p99 ≈ p95).

So the metric answers "how reliable is it, and what does its obfuscation / loss-recovery cost in latency" — e.g. TYPHOON delivers like the other UDP transports yet keeps a flat tail (its decoys / fake-headers add no measurable latency penalty), unlike DAITA. It is deliberately not a throughput benchmark: the shared capture point and best-effort UDP's lack of congestion control make bulk throughput an artifact across transport classes (an earlier bulk-transfer design was abandoned for exactly this reason — see the git history). For the TYPHOON implementation's raw speed on loopback, free of these network artifacts, see Part 4 (cargo bench).

Per-pcap metrics (stats.json)

Computed separately per direction (c2s, s2c, all). Packet sizes are transport-payload bytes (UDP payload or TCP segment data) — IP/UDP/TCP header bytes are excluded so transport overhead doesn't leak into protocol stats.

Metric Meaning
packet_count, byte_count, transmission_time_s Volume and duration
packet_size.{mean,std,min,max,p5…p99,entropy} Size distribution
iat_ms.{mean,std,p5…p99,entropy} Inter-arrival-time distribution
entropy.{all,handshake,data} Shannon entropy of payload bytes (8.0 ≈ random)
burstiness std(IAT) / mean(IAT) — temporal clustering
size_regularity n_distinct_sizes / n_packets
overhead_ratio, goodput_efficiency Wire bytes vs. user data
direction_asymmetry c2s_bytes / s2c_bytes
first_n_sizes[100], first_n_iats[100] First-100 direction-signed sequences (for ML)
hs_duration_s, hs_pkt_count, hs_byte_frac Handshake window (when sniffer configured)

Part 1 plots (under artifacts/<pipeline_id>/)

  • self_compare/default_self_compare.pdf — overlaid size + IAT CDFs across N runs of identical config. Look for tight bands.
  • traffic_compare/default_traffic_compare.pdf — same CDFs per scenario. Look for clearly separated bands.
  • use_case_compare/use_case_compare.pdf — per-PROTOCOL.md-use-case profiles side by side.
  • flow_plots/run_<id>_pcap_flow.pdf — stacked-bar of every wire packet showing fake-header / trailer / body composition. Useful to verify per-flow knobs.

Part 2 plots (under artifacts/<pipeline_id>/proto_compare/)

  • run_<id>_proto_compare.pdf — six panels: (A) size CDF, (B) IAT CDF, (C) per-packet RTT vs delivery scatter, (D) overhead bars, (E) byte entropy by phase, (F) normalised heatmap.
  • run_<id>_handshake.pdf — handshake duration / packet count / byte fraction across protocols.
  • run_<id>_compare_table.md — one row per protocol; quick-glance ranking by any column.

Part 3 outputs

  • background-blending prints the confident-blend fraction (Test C) — the share of TYPHOON flows the open-world classifier labels as a concrete background class with high confidence. Higher = less distinguishable. It evaluates every Barradas classifier (rf/dt/xgb) with k-fold out-of-fold scoring and reports them all, headlining the adversary-strongest (the one catching the most TYPHOON at the shared ~1 % background false-positive point) — so the reported worst case tracks whichever classifier is best on the data, not a hard-coded one. --classifier narrows the set.
  • background-detectability reports per-test scores (Tests A, B, D, E, F above). Each test answers a distinct ML setup; treat them as complementary not redundant.
  • background-distplot overlays the actual TYPHOON size/IAT distributions on each background class — visual check of where TYPHOON differs.

Settings overrides

Every TYPHOON protocol constant (handshake length, decoy rates, fake header bounds, …) can be overridden via environment variables on the TYPHOON containers — these are passed straight through to the Rust crate's SettingsBuilder. Useful to A/B-test parameter changes against the same corpus.