HFT Orderbook Data Collector

Streams Bybit linear-perpetual orderbook data via WebSocket and archives it as hourly gzip-compressed NDJSON files on Google Cloud Storage.

Technical Highlights

• 20ms updates at depth 50 — subscribes to Bybit's orderbook.50.<SYMBOL> stream, whose level-50 feed has a 20ms push frequency.
• 100k buffered channel per symbol — isolates WebSocket ingestion from storage writes and drops instead of blocking if the queue is exhausted.
• Nanosecond receive timestamps — wraps every raw Bybit payload with time.Now().UnixNano() immediately after receipt.
• Hourly gzip NDJSON partitions — writes one UTC-hour object per symbol under a deterministic GCS prefix.
• EC2 + systemd deployment — includes a unit file, automatic restart policy, journald logging, and a VM setup helper.
• Write-only scoped identity — supports a dedicated GCS service account restricted to object creation in the target bucket.
• Graceful shutdown — handles Ctrl+C and SIGTERM, flushes each writer's pending partial-hour buffer, and waits for all writers before exiting.

Architecture

Bybit public WebSocket (`orderbook.50`, 20ms)
        │
        ▼
Feed goroutine ──► receive timestamp + raw payload
        │
        ▼
buffered channel (100,000 messages per symbol)
        │
        ▼
Writer goroutine ──► hourly buffer ──► gzip NDJSON ──► GCS
        ▲
        │
SIGINT / SIGTERM ──► cancel ──► flush pending buffer ──► exit

One feed + writer goroutine pair runs per symbol. All symbols share a single GCS client.

Output path pattern:

gs://<bucket>/bybit/linear/<SYMBOL>/orderbook<DEPTH>/<YYMMDDH2H2>.ndjson.gz

Example: gs://crypto-hft-data/bybit/linear/BTCUSDT/orderbook50/26052215.ndjson.gz

---

Quick Start

1. Clone

git clone https://github.com/kiwicabbage/HFT-crypto-data-fetch.git
cd HFT-crypto-data-fetch

2. Prerequisites

• Go 1.22+
• A Google Cloud Storage bucket
• A dedicated GCS service account with Storage Object Creator on that bucket

3. Configure

cp .env.example .env
# Edit .env and fill in GCS_BUCKET, SYMBOLS, GOOGLE_APPLICATION_CREDENTIALS

4. Build & Run (local)

# Source your env vars
export $(grep -v '^#' .env | xargs)

# Build and run
make run

5. Build for Linux / Deploy to AWS EC2

# Cross-compile for Linux (from Mac)
make build-linux

# Upload to EC2
make deploy VM=ubuntu@<EC2-IP> KEYPATH=~/.ssh/your-key.pem

# SSH in and restart the service
ssh -i ~/.ssh/your-key.pem ubuntu@<EC2-IP>
sudo systemctl restart hft-collector

---

CLI Flags

Flag	Default	Description
--symbols	BTCUSDT	Comma-separated list of symbols, e.g. BTCUSDT,ETHUSDT,SOLUSDT
--depth	50	Orderbook depth (1 / 50 / 200 / 1000)
--bucket	$GCS_BUCKET	GCS bucket name
--key	$GOOGLE_APPLICATION_CREDENTIALS	Path to service-account JSON key

All flags can also be set via environment variables (see .env.example).

---

Performance Profile

Component	Setting	Effect
Bybit feed	Depth 50, 20ms push frequency	Up to 50 feed updates per second when the book changes
Ingestion queue	100,000 messages per symbol	Absorbs temporary writer or network-storage stalls
Receive clock	Nanoseconds (recv_ns)	Preserves a high-resolution local arrival timestamp
Storage partition	One gzip NDJSON object per UTC hour and symbol	Keeps files compressed, append-friendly, and time-partitioned
Backpressure	Non-blocking enqueue	Protects the WebSocket reader; logs and drops when the queue is full

The 20ms figure is Bybit's advertised push frequency for level-50 orderbook streams, not a guaranteed end-to-end storage latency.

---

Makefile Targets

Command	Description
make build	Compile for your local OS
make build-linux	Cross-compile for Linux amd64 (for AWS EC2)
make run	Build and run locally (requires .env to be sourced)
make deploy VM=... KEYPATH=...	Build for Linux and upload to EC2 via scp
make clean	Remove compiled binaries

---

EC2 Deployment (systemd)

The deploy/ directory contains:

• hft-collector.service — systemd unit file. Edit ExecStart to set your symbols.
• setup-vm.sh — one-time setup script for a fresh Ubuntu EC2 instance.

Least-privilege GCS access

Create a dedicated service account for this collector and grant roles/storage.objectCreator only on the destination bucket. That role allows the collector to create objects without granting object read, delete, or overwrite access. Keep the downloaded credential outside the repository, restrict its file permissions to 600, and point GOOGLE_APPLICATION_CREDENTIALS at it. The repository ignores .env, *.key, and JSON credential files.

First-time setup

# From your Mac — copy files to EC2
scp -i ~/.ssh/key.pem deploy/hft-collector.service ubuntu@<EC2-IP>:~/
scp -i ~/.ssh/key.pem hft-collector-key.json ubuntu@<EC2-IP>:~/
scp -i ~/.ssh/key.pem collector-linux ubuntu@<EC2-IP>:~/collector

# SSH in and install the service
ssh -i ~/.ssh/key.pem ubuntu@<EC2-IP>
chmod +x ~/collector
chmod 600 ~/hft-collector-key.json
sudo cp ~/hft-collector.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now hft-collector

Useful commands on EC2

sudo systemctl status hft-collector       # check status
journalctl -u hft-collector -f            # tail live logs
sudo systemctl restart hft-collector      # restart after update

---

File Layout

.
├── cmd/
│   └── collector/
│       └── main.go            # Entry point — parses flags, wires goroutines
├── internal/
│   ├── bybit/
│   │   └── orderbook.go       # WebSocket feed with auto-reconnect & heartbeat
│   └── writer/
│       └── gcs.go             # Hourly gzip NDJSON writer to GCS
├── deploy/
│   ├── hft-collector.service  # systemd unit file
│   └── setup-vm.sh            # One-time EC2 setup script
├── .env.example               # Copy to .env and fill in secrets
├── Makefile
├── go.mod
└── go.sum

---

Data Format

Each line in the .ndjson.gz file is a JSON object:

{"recv_ns": 1716394815123456789, "raw": {"topic":"orderbook.50.BTCUSDT","ts":1716394815120,...}}

Field	Description
recv_ns	Nanosecond wall-clock timestamp when the message was received
raw	Verbatim JSON payload from Bybit — snapshot (type:"snapshot") or delta (type:"delta")