Skip to content

Commit f64696d

Browse files
BlueSkyXNBlueSkyXN
committed
fix(hfs): pin bucket storage sampler client
1 parent 8ac080b commit f64696d

4 files changed

Lines changed: 59 additions & 16 deletions

File tree

‎docs/operations.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -193,10 +193,10 @@ scripts/sample-hf-bucket-storage.sh \
193193

194194
| 字段 | 含义 |
195195
| --- | --- |
196-
| `info_size` | `hf buckets info` 返回的 bucket 账面 size。 |
197-
| `info_total_files` | `hf buckets info` 返回的 total files。 |
198-
| `visible_sum` | `hf buckets list -R` 当前 visible file size 求和。 |
199-
| `visible_files` | `hf buckets list -R` 当前 visible file 数。 |
196+
| `info_size` | pinned `HfApi.bucket_info()` 返回的 bucket 账面 size。 |
197+
| `info_total_files` | pinned `HfApi.bucket_info()` 返回的 total files。 |
198+
| `visible_sum` | pinned module CLI `buckets list --recursive --format json` 当前 visible file size 求和。 |
199+
| `visible_files` | pinned module CLI `buckets list --recursive --format json` 当前 visible file 数。 |
200200
| `drift_bytes` | `info_size - visible_sum`。 |
201201
| `ops_visible_sum` | 可选 `/_ops/storage` 返回的容器内 visible bytes。 |
202202
| `largest_visible_file` | HF visible tree 中最大的当前文件。 |

‎docs/source-walkthrough.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -427,7 +427,7 @@ ops 鉴权支持三种方式：
427427
- `hfs/start.sh` 的 Secret 校验、公开 URL 推导、Console redirect URL、ops/admin service 和 Nginx 配置预检。
428428
- `hfs/nginx.conf` 的 `7860` 监听、`/_ops/`、`/_admin/`、`/console/` 子路径、S3 根路径和 iframe header。
429429
- `hfs/ops_service.py` 的 `/_ops/storage` 路由和 storage metrics。
430-
- `scripts/sample-hf-bucket-storage.sh` 的存在性、可执行位、Bash 语法和 `hf`/`python3` 只读采样约束。
430+
- `scripts/sample-hf-bucket-storage.sh` 的存在性、可执行位、Bash 语法和 pinned `HfApi`/module CLI 只读采样约束。
431431
- `LICENSE` 是否仍是 AGPL-3.0。
432432
- 如果本机安装了 `nginx`，运行 `nginx -t -c "$PWD/hfs/nginx.conf"`。
433433

@@ -465,10 +465,10 @@ scripts/sample-hf-bucket-storage.sh \
465465
--ops-url https://blueskyxn-librefs-hfs.hf.space/_ops
466466
```
467467

468-
脚本会调用：
468+
脚本固定验证 `huggingface_hub==1.5.0` 与 `click==8.3.1`，然后调用：
469469

470-
- `hf buckets info "$bucket" --json`
471-
- `hf buckets list "$bucket" -R --json`
470+
- `HfApi().bucket_info("$bucket")`，生成只包含 bucket metadata 的结构化 JSON
471+
- `python3 -m huggingface_hub.cli.hf buckets list "$bucket" --recursive --format json`
472472
- 可选 `curl -H "X-Ops-Token: $OPS_TOKEN" "$ops_url/storage?format=json"`
473473

474474
输出字段包括 `timestamp_utc`、`info_size`、`info_total_files`、`visible_sum`、`visible_files`、`drift_bytes`、`ops_visible_sum` 和 `largest_visible_file`。脚本用 `python3` 解析 JSON，不依赖 `jq`，也不会打印 `OPS_TOKEN`。

‎scripts/sample-hf-bucket-storage.sh‎

Lines changed: 38 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22
set -Eeuo pipefail
33

44
DEFAULT_BUCKET="BlueSkyXN/librefs-hfs-data"
5+
EXPECTED_HF_HUB_VERSION="1.5.0"
6+
EXPECTED_CLICK_VERSION="8.3.1"
57
BUCKET="${HF_BUCKET_ID:-$DEFAULT_BUCKET}"
68
OPS_URL="${OPS_URL:-}"
79
INTERVAL=0
@@ -83,13 +85,28 @@ if [[ "$COUNT" == "0" && "$INTERVAL" == "0" ]]; then
8385
exit 2
8486
fi
8587

86-
if ! command -v hf >/dev/null 2>&1; then
87-
echo "hf CLI is required." >&2
88+
if ! command -v python3 >/dev/null 2>&1; then
89+
echo "python3 is required." >&2
8890
exit 1
8991
fi
9092

91-
if ! command -v python3 >/dev/null 2>&1; then
92-
echo "python3 is required." >&2
93+
if ! python3 - "$EXPECTED_HF_HUB_VERSION" "$EXPECTED_CLICK_VERSION" <<'PY'
94+
import importlib.metadata
95+
import sys
96+
97+
expected_hub, expected_click = sys.argv[1:]
98+
try:
99+
actual_hub = importlib.metadata.version("huggingface_hub")
100+
actual_click = importlib.metadata.version("click")
101+
except importlib.metadata.PackageNotFoundError as exc:
102+
raise SystemExit(f"required Python package is missing: {exc.name}") from exc
103+
if actual_hub != expected_hub or actual_click != expected_click:
104+
raise SystemExit(
105+
"storage sampler requires "
106+
f"huggingface_hub=={expected_hub} and click=={expected_click}"
107+
)
108+
PY
109+
then
93110
exit 1
94111
fi
95112

@@ -124,8 +141,23 @@ sample_once() {
124141
local list_json="$TMP_DIR/list.json"
125142
local ops_json="$TMP_DIR/ops.json"
126143

127-
hf buckets info "$BUCKET" --json >"$info_json"
128-
hf buckets list "$BUCKET" -R --json >"$list_json"
144+
python3 - "$BUCKET" >"$info_json" <<'PY'
145+
import json
146+
import sys
147+
148+
from huggingface_hub import HfApi
149+
150+
info = HfApi().bucket_info(sys.argv[1])
151+
print(json.dumps({
152+
"id": info.id,
153+
"private": info.private,
154+
"createdAt": info.created_at.isoformat(),
155+
"size": info.size,
156+
"totalFiles": info.total_files,
157+
}, sort_keys=True))
158+
PY
159+
python3 -m huggingface_hub.cli.hf buckets list \
160+
"$BUCKET" --recursive --format json >"$list_json"
129161

130162
if [[ -n "$OPS_URL" ]]; then
131163
curl -fsS -H "X-Ops-Token: ${OPS_TOKEN}" "$(ops_endpoint)" >"$ops_json"

‎scripts/validate-contract.sh‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -206,9 +206,20 @@ require_pattern scripts/smoke-s3-curl.sh 'Authorization: Bearer' 'private candid
206206
require_pattern scripts/smoke-s3-curl.sh 'Refusing to use bucket' 'S3 smoke test must refuse existing buckets'
207207

208208
check "storage sampler script contract" test -f scripts/sample-hf-bucket-storage.sh
209-
require_pattern scripts/sample-hf-bucket-storage.sh 'hf buckets info "\$BUCKET" --json' 'storage sampler must read HF bucket accounting'
210-
require_pattern scripts/sample-hf-bucket-storage.sh 'hf buckets list "\$BUCKET" -R --json' 'storage sampler must read HF bucket visible tree'
209+
require_pattern scripts/sample-hf-bucket-storage.sh '^EXPECTED_HF_HUB_VERSION="1\.5\.0"$' 'storage sampler must pin huggingface_hub 1.5.0'
210+
require_pattern scripts/sample-hf-bucket-storage.sh '^EXPECTED_CLICK_VERSION="8\.3\.1"$' 'storage sampler must pin the module CLI click runtime'
211+
require_pattern scripts/sample-hf-bucket-storage.sh 'HfApi\(\)\.bucket_info\(sys\.argv\[1\]\)' 'storage sampler must use structured HfApi bucket accounting'
212+
require_pattern scripts/sample-hf-bucket-storage.sh 'python3 -m huggingface_hub\.cli\.hf buckets list' 'storage sampler must use the pinned module CLI for the visible tree'
213+
require_pattern scripts/sample-hf-bucket-storage.sh '"\$BUCKET" --recursive --format json' 'storage sampler must request recursive JSON visible-tree output'
211214
require_pattern scripts/sample-hf-bucket-storage.sh 'OPS_TOKEN' 'storage sampler must use OPS_TOKEN only for optional ops endpoint access'
215+
if grep -Eq '(^|[^[:alnum:]_.-])hf[[:space:]]+buckets[[:space:]]+(info|list)([[:space:]]|$)' scripts/sample-hf-bucket-storage.sh; then
216+
echo "Contract check failed: storage sampler must not use the unpinned hf console entrypoint" >&2
217+
exit 1
218+
fi
219+
if grep -Eq 'buckets[[:space:]]+info[^[:cntrl:]]*--(json|format)' scripts/sample-hf-bucket-storage.sh; then
220+
echo "Contract check failed: storage sampler must not claim buckets info has JSON output" >&2
221+
exit 1
222+
fi
212223

213224
check "nginx routing contract" require_pattern hfs/nginx.conf 'listen 7860;' 'Nginx must listen on HF app port 7860'
214225
require_pattern hfs/nginx.conf 'location = /console' 'Nginx must normalize /console'

0 commit comments

Comments
 (0)