GreptimeDB download: 5-min Client.Timeout causes infinite retry loop on slow connections (150MB > 5min)

[wangdengwu]

Summary

On a fresh install, tma1-server can get stuck in an infinite GreptimeDB download → 5‑minute‑timeout → retry loop on slower connections, never reaching a healthy state. The dashboard never comes up and the only signal is a repeating failed to install greptimedb ... context deadline exceeded line in the log.

Environment

• TMA1 v0.2.0-alpha7, macOS (darwin/arm64)
• GreptimeDB v1.1.1, asset greptime-darwin-arm64-v1.1.1.tar.gz is ~150 MB (content-length: 150472626)
• Connection throughput during repro: ~27 MB/min

Root cause

server/internal/install/install.go:45

downloadClient = &http.Client{Timeout: 5 * time.Minute}

This Timeout is the total deadline for the whole request, including streaming the response body in downloadAndExtract (install.go:~348):

resp, err := downloadClient.Get(url)
...
_, err = io.Copy(dst, resp.Body)   // <- the 5-min clock is still ticking here

A 150 MB tarball at ~27 MB/min needs ~5.5 min — just over the cap — so io.Copy dies mid‑stream with context deadline exceeded, the partial download is discarded, and EnsureGreptimeDB retries from scratch with no Range/resume and no backoff (observed retry ~2 s after the failure). On any connection where the download is slower than 150 MB / 5 min, this loops forever.

Observed log

18:21:24 INFO  downloading greptimedb version=v1.1.1
18:26:24 ERROR failed to install greptimedb err="install: download .../greptime-darwin-arm64-v1.1.1.tar.gz: context deadline exceeded (Client.Timeout or context cancellation while reading body)"
18:26:26 INFO  downloading greptimedb version=v1.1.1     # immediate retry, restarts at 0 bytes

Steps to reproduce

1. Throttle the network to < ~30 MB/min (or repro on any moderately slow link).
2. curl -fsSL https://tma1.ai/install.sh | TMA1_VERSION=v0.2.0-alpha7 bash
3. Watch ~/Library/Logs/tma1-server.log — the download fails at the 5‑min mark and retries indefinitely; curl http://localhost:14318/health never returns ok.

Impact

First‑run experience hard‑fails for anyone on a slower connection, with no actionable message (just a repeating timeout). Likely to grow as the GreptimeDB binary gets larger.

Workaround

Download the tarball out of band, drop greptime into ~/.tma1/bin/greptime (chmod +x), write v1.1.1 to ~/.tma1/bin/.version, and restart the service — EnsureGreptimeDB then sees the binary and skips the download.

Suggested fixes (any subset)

1. Don't cap body read with a wall‑clock total timeout. Drop Client.Timeout for the download client and instead enforce a stall timeout (idle/read deadline) so a slow‑but‑progressing transfer isn't killed. e.g. http.Transport{ResponseHeaderTimeout: ...} + a stall‑detecting reader, or a context you reset on progress.
2. Resumable download via HTTP Range so a retry continues from the partial file instead of restarting at 0.
3. Backoff + attempt cap between retries, with a clear terminal error after N attempts instead of an infinite silent loop.
4. Progress / clearer messaging (bytes downloaded, total) so a stuck/slow download is diagnosable from the log.

[killme2008]

Thanks @wangdengwu — the root-cause analysis is exactly right: the 5-minute Client.Timeout is a total deadline that includes the body read, so a ~150 MB tarball on a sub-30 MB/min link gets cut mid-stream and the fallback restarts from zero.

For the fix we'd rather keep it minimal than take the full resumable-download route in #58 — Range + partial-file lifecycle + self-healing is a lot of surface for a fallback path that the recommended install.sh (which pre-fetches GreptimeDB via the official script) doesn't normally hit. Your suggested fix #1 is enough on its own:

• Drop the wall-clock Client.Timeout on the download client and bound the connect/header phase with Transport{ResponseHeaderTimeout: 60s} instead. A slow-but-progressing transfer is no longer guillotined, while a dead endpoint still fails fast.
• One thing to watch: a bare &http.Transport{...} drops proxy support. Keep Proxy: http.ProxyFromEnvironment so http_proxy / https_proxy still work — the current default client gets this for free via http.DefaultTransport, and fix(install): resumable GreptimeDB download instead of restart-from-zero #58 lost it.

If you'd like to revise the PR down to that minimal change, we'd be glad to take it. Resume/backoff can follow later if it proves necessary, but the timeout swap alone fixes the slow-link loop.

In the meantime we've added Troubleshooting entries to the setup skill for the network failure modes:

• http_proxy / https_proxy for restricted/blocked networks,
• TMA1_VERSION pinning when the GitHub tag lookup fails,
• and the out-of-band drop-in you described (~/.tma1/bin/greptime + .version) works today for a purely slow link.

Closing #58 in favor of the slimmed-down change — thanks for the detailed report and PR.