Skip to content

walkthrough.md

Latest commit

 

History

History
259 lines (186 loc) · 6.36 KB

walkthrough.md

File metadata and controls

259 lines (186 loc) · 6.36 KB
Raw

HFT Collector — Setup & Run Guide

What was built

26HFTdataFetch/
├── go.mod / go.sum
├── cmd/collector/main.go          ← entrypoint
└── internal/
    ├── bybit/orderbook.go         ← Bybit WebSocket feed
    └── writer/gcs.go              ← hourly GCS writer

Data flow:

Bybit WebSocket (orderbook.50.BTCUSDT, 20ms updates)
  └─▶ channel (100k-message buffer)
        └─▶ GCS Writer
              └─▶ gs://crypto-hft-data/bybit/linear/BTCUSDT/orderbook50/26052215.ndjson.gz

Each file = one UTC hour of raw orderbook events, gzip-compressed NDJSON.
Each line: {"recv_ns": <nanoseconds>, "raw": { ...bybit message... }}

Part 1 — Google Cloud Setup (do this once)

Step 1 — Install the Google Cloud CLI

Open Terminal and run:

brew install --cask google-cloud-sdk

Verify it works:

gcloud --version

Step 2 — Log in to your Google account

gcloud auth login

A browser window will open. Log in with your Google account.
After login, come back to Terminal and continue.

Step 3 — Set your project

# List your projects to find the right one
gcloud projects list

# Set the active project (replace YOUR_PROJECT_ID)
gcloud config set project YOUR_PROJECT_ID

Note

If you don't have a project yet, create one at https://console.cloud.google.com/projectcreate

Step 4 — Enable the Cloud Storage API

gcloud services enable storage.googleapis.com

Step 5 — Create the GCS bucket

gcloud storage buckets create gs://crypto-hft-data \
  --location=ASIA-EAST1 \
  --uniform-bucket-level-access

Tip

Use the region closest to you. Options: ASIA-EAST1 (Taiwan), ASIA-SOUTHEAST1 (Singapore), US-CENTRAL1 (Iowa). Closer = lower latency when uploading.

Step 6 — Create a Service Account

A service account is like a robot user that your program logs in as. It gets a key file (JSON) that the program uses instead of your personal password.

# Create the service account
gcloud iam service-accounts create hft-collector \
  --display-name="HFT Data Collector"

Step 7 — Grant the service account permission to write to the bucket

gcloud storage buckets add-iam-policy-binding gs://crypto-hft-data \
  --member="serviceAccount:hft-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectCreator"

Important

Replace YOUR_PROJECT_ID with your actual project ID from Step 3.

Step 8 — Download the key file

gcloud iam service-accounts keys create ~/hft-collector-key.json \
  --iam-account=hft-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com

This downloads a file called hft-collector-key.json to your home directory.

Caution

Keep this file private. Never commit it to Git or share it. It acts as your program's password to GCS. Add it to .gitignore immediately.

Add to .gitignore:

echo "*.json" >> /Users/superstone/code/26HFTdataFetch/.gitignore
echo "hft-collector-key.json" >> ~/.gitignore_global

Part 2 — Run the Collector

Step 9 — Build the binary

cd /Users/superstone/code/26HFTdataFetch
go build -o collector ./cmd/collector

Step 10 — Run it

GOOGLE_APPLICATION_CREDENTIALS=~/hft-collector-key.json \
GCS_BUCKET=crypto-hft-data \
./collector --symbol BTCUSDT --depth 50

Or pass the key path as a flag:

./collector \
  --bucket crypto-hft-data \
  --key ~/hft-collector-key.json \
  --symbol BTCUSDT \
  --depth 50

Expected startup output:

=== HFT Data Collector ===
Exchange : Bybit (linear perpetual)
Symbol   : BTCUSDT
Depth    : 50
Bucket   : gs://crypto-hft-data
Auth     : service-account key /Users/you/hft-collector-key.json
[bybit] connected — subscribing to orderbook.50.BTCUSDT

Then every hour on the hour:

[writer] flushing 180342 messages → gs://crypto-hft-data/bybit/linear/BTCUSDT/orderbook50/26052215.ndjson.gz
[writer] ✓ bybit/linear/BTCUSDT/orderbook50/26052215.ndjson.gz — 18432.7 KB

Press Ctrl+C to stop. The current partial-hour buffer will be flushed to GCS before exit.

Part 3 — Verify Your Data

List files in your bucket

gcloud storage ls gs://crypto-hft-data/bybit/linear/BTCUSDT/orderbook50/

Download and inspect a file

gcloud storage cp \
  gs://crypto-hft-data/bybit/linear/BTCUSDT/orderbook50/26052215.ndjson.gz \
  /tmp/test.ndjson.gz

zcat /tmp/test.ndjson.gz | head -3

Expected output (one JSON line per event):

{"recv_ns":1716390601234567890,"raw":{"topic":"orderbook.50.BTCUSDT","type":"snapshot","ts":1716390601200,"data":{"s":"BTCUSDT","b":[["104500.00","1.234"],...],"a":[["104501.00","0.567"],...],"u":1234567,"seq":9876543},"cts":1716390601199}}
{"recv_ns":1716390601254321098,"raw":{"topic":"orderbook.50.BTCUSDT","type":"delta","ts":1716390601250,...}}

File Naming Reference

UTC Hour GCS Object Name
00:00 – 00:59 …/26052200.ndjson.gz
01:00 – 01:59 …/26052201.ndjson.gz
15:00 – 15:59 …/26052215.ndjson.gz
23:00 – 23:59 …/26052223.ndjson.gz

Format: YYMMDD + two-digit UTC hour (00–23).

Quick Reference — All Terminal Commands in Order

# 1. Install gcloud
brew install --cask google-cloud-sdk

# 2. Login
gcloud auth login

# 3. Set project (replace YOUR_PROJECT_ID)
gcloud config set project YOUR_PROJECT_ID

# 4. Enable API
gcloud services enable storage.googleapis.com

# 5. Create bucket
gcloud storage buckets create gs://crypto-hft-data --location=ASIA-EAST1 --uniform-bucket-level-access

# 6. Create service account
gcloud iam service-accounts create hft-collector --display-name="HFT Data Collector"

# 7. Grant bucket write permission
gcloud storage buckets add-iam-policy-binding gs://crypto-hft-data \
  --member="serviceAccount:hft-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectCreator"

# 8. Download key file
gcloud iam service-accounts keys create ~/hft-collector-key.json \
  --iam-account=hft-collector@YOUR_PROJECT_ID.iam.gserviceaccount.com

# 9. Build
cd /Users/superstone/code/26HFTdataFetch
go build -o collector ./cmd/collector

# 10. Run
GOOGLE_APPLICATION_CREDENTIALS=~/hft-collector-key.json GCS_BUCKET=crypto-hft-data \
  ./collector --symbol BTCUSDT --depth 50