Etihad air-bounds scraper (Go)

Scrapes the Etihad flight-search JSON API (https://api-des.etihad.com/airlines/EY/v2/search/air-bounds) from request-based Go, bootstrapping a valid session by solving the Imperva reese84 anti-bot challenge with the Hyper Solutions API and a bogdanfinn tls-client browser-grade TLS fingerprint.

Anti-bot analysis (from the recorded session)

digital.etihad.com sits behind both Akamai Bot Manager and Imperva, but the JSON API on api-des.etihad.com is guarded only by Imperva reese84 — it authorizes each call with an x-d-token header (the reese84 token) plus an OAuth Bearer. Akamai is not on the API path, so we never touch the Akamai sensor flow.

The reese84 protection here is the "dynamic" (Pardon Our Interruption) variant with a Proof-of-Work step. The full bootstrap:

1. GET /book/search?… → interruption HTML + Imperva session cookies (visid_incap_*, incap_ses_*). The obfuscated reese84 script path is embedded in the HTML and rotates every load, so it is parsed dynamically, never hardcoded.
2. GET the reese84 challenge script (large, obfuscated JS).
3. POST {"f":"gpc"} to the sensor endpoint → Proof-of-Work string.
4. Hyper Solutions GenerateReese84Sensor (script + PoW + UA + IP) → sensor payload.
5. POST the sensor → {"token":"3:…","renewInSec":…}. That token is the x-d-token.

Then the API calls:

6. POST /v1/security/oauth2/token/initialization (client-credentials + x-d-token) → OAuth Bearer.
7. POST /airlines/EY/v2/search/air-bounds (Bearer + x-d-token) → flight results.

Why the details matter

• TLS fingerprint: a latest-Chrome profile with randomized extension order, HTTP/3 disabled. Matches the recorded JA4 t13d1516h2_….
• Header order: every request replays Chrome's exact wire header order (captured with powhttp) via fhttp's HeaderOrderKey — the top request-side fingerprint signal.
• IP consistency: the IP reported to Hyper (GET https://ip.hypersolutions.co/ip) must equal the IP the target sees. Use a sticky proxy, never rotating.
• Session: one cookie jar, one UA, one IP for the whole flow.

Usage

export HYPER_API_KEY="<your-hyper-solutions-key>"   # required
export PROXY_URL="http://user:pass@host:port"        # strongly recommended (sticky IP)

go run .

Progress is logged to stderr; the raw air-bounds JSON is printed to stdout.

Configuration (environment variables)

Var	Default	Meaning
HYPER_API_KEY	— (required)	Hyper Solutions API key
PROXY_URL	none (direct IP)	Sticky HTTP proxy
OUTBOUND_IP	auto via Hyper /ip	Skip the IP lookup if you already know your egress IP
EY_ORIGIN / EY_DEST	CDG / ICN	Route
EY_DATE	2026-07-17	Departure date (YYYY-MM-DD)
EY_FLOW	AWARD	AWARD (miles) or REVENUE (cash)
EY_MARKET / EY_ORIGIN_COUNTRY	AU / FR	POS context for the OAuth fact blob

Layout

File	Responsibility
main.go	Wires the flow: IP → reese84 → OAuth → search
config.go	Env config + browser identity
tlsclient.go	bogdanfinn tls-client + header-ordered request helper
reese84.go	Imperva reese84 dynamic bootstrap (the anti-bot part)
api.go	OAuth token exchange + air-bounds search + IP lookup
util.go	Logging / JSON helpers

Notes

• The reese84 token has a renewInSec (~850s); for long-running scraping, re-run the bootstrap before it expires.
• oauthClientID / oauthClientSecret are the public SPA credentials embedded in Etihad's front-end (captured from the session), not user secrets.
• The air-bounds request body sends only the core anonymous search (fare families, itinerary, travelers). The recorded session additionally carried a frequentFlyer / corporateCodes block tied to a logged-in loyalty account — omitted here.