Skip to content

Commit cb63c45

Browse files
authored
feat(dashboard): add PyPI mirror picker and speed tests for network sources (#10657)
* feat(dashboard): flatten GitHub proxy picker and test all options - Move the connectivity test button next to the section title instead of the "use GitHub proxy" radio row. - Drop the nested radio group so the proxy URLs sit at the same level as "no proxy", giving one flat list of choices. - Test every option at once, including the direct (no proxy) route and a custom URL, and show the result per option. - Let the ghproxy test endpoint accept an empty proxy_url to probe GitHub directly, which is what the "no proxy" option needs. - Remove the now unused proxySelector.useProxy i18n key. * feat(dashboard): add PyPI index picker with presets and connectivity test * feat(dashboard): list official PyPI index as a preset * feat(dashboard): add real download speed test to source pickers * fix(dashboard): keep custom source input visible and aligned with presets * fix(dashboard): refine source picker wording and mirror order * fix(dashboard): remember PyPI custom index across reloads * fix(dashboard): cap PyPI index page read in download speed test measure_pypi_index_speed read the whole /pip/ page with response.text() before applying any size limit, so an oversized or endless page from a configured or caller-supplied index could exhaust dashboard memory. Stream the body with a hard 8 MB budget and abort with a StatServiceError as soon as it is exceeded. * fix(dashboard): parse PyPI index links with an HTML parser The speed test extracted wheels with a regex for lowercase, double-quoted href attributes, so a valid simple index using single quotes, unquoted values, or uppercase HREF reported "No pip wheel found on this index". Collect anchor hrefs with html.parser.HTMLParser instead, which handles attribute quoting and casing, and strip the "#sha256=..." fragment explicitly. * style(dashboard): apply ruff format to stat service `ruff format --check` flagged stat_service.py on this branch: a missing blank line after StatServiceError and an over-long wheel filter from the earlier speed-test work. Reformat with the pinned ruff 0.15.22 so CI is green again. * feat(dashboard): merge network connectivity and speed tests into one button The network page exposed separate connectivity and speed buttons. Running them one after another is the normal flow, so a single 测试连通性 button now probes the latency of every source first and only measures throughput once all latency results are in. Both selectors share the same two-stage flow with a single loading state, and the unused testSpeed locale keys are dropped from all four bundles. * feat(dashboard): test proxy sources one by one from the selected route - probe each route sequentially so latency and speed results appear route by route instead of all at once - start the sweep at the currently selected route, then keep the listed order of the remaining routes - show a per-route progress indicator while its probe is running * feat(dashboard): add Ctrl/Cmd and Shift modes to the connectivity test - Ctrl/Cmd + click re-tests only the selected route and keeps the other results, Shift + click probes every route at once - run all latency probes before any throughput probe in the parallel mode so a saturating download cannot skew a sibling's latency - add a tooltip listing the three click modes in all four locales --------- Co-authored-by: XTsat <XTsat@users.noreply.github.com>
1 parent bd084a1 commit cb63c45

14 files changed

Lines changed: 1186 additions & 105 deletions

File tree

‎astrbot/dashboard/api/stats.py‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,11 @@
44

55
from astrbot.dashboard.async_utils import run_maybe_async
66
from astrbot.dashboard.responses import ApiError, ok
7-
from astrbot.dashboard.schemas import GhProxyTestRequest, StorageCleanupRequest
7+
from astrbot.dashboard.schemas import (
8+
GhProxyTestRequest,
9+
PypiIndexTestRequest,
10+
StorageCleanupRequest,
11+
)
812
from astrbot.dashboard.services.stat_service import StatService, StatServiceError
913

1014
from .auth import AuthContext, require_dashboard_user, require_scope
@@ -104,6 +108,33 @@ async def test_ghproxy_connection(
104108
return await _run(service.test_ghproxy_connection(payload.proxy_url))
105109

106110

111+
@router.post("/stats/pypi-index/test")
112+
async def test_pypi_index_connection(
113+
payload: PypiIndexTestRequest,
114+
_auth: AuthContext = Depends(require_system_scope),
115+
service: StatService = Depends(get_service),
116+
):
117+
return await _run(service.test_pypi_index_connection(payload.index_url))
118+
119+
120+
@router.post("/stats/pypi-index/speed")
121+
async def measure_pypi_index_speed(
122+
payload: PypiIndexTestRequest,
123+
_auth: AuthContext = Depends(require_system_scope),
124+
service: StatService = Depends(get_service),
125+
):
126+
return await _run(service.measure_pypi_index_speed(payload.index_url))
127+
128+
129+
@router.post("/stats/ghproxy/speed")
130+
async def measure_ghproxy_speed(
131+
payload: GhProxyTestRequest,
132+
_auth: AuthContext = Depends(require_system_scope),
133+
service: StatService = Depends(get_service),
134+
):
135+
return await _run(service.measure_ghproxy_speed(payload.proxy_url))
136+
137+
107138
@router.get("/changelogs")
108139
async def list_changelog_versions(
109140
_auth: AuthContext = Depends(require_system_scope),

‎astrbot/dashboard/schemas.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -222,6 +222,10 @@ class GhProxyTestRequest(BaseModel):
222222
proxy_url: str | None = None
223223

224224

225+
class PypiIndexTestRequest(BaseModel):
226+
index_url: str | None = None
227+
228+
225229
class OpenApiChatRequest(OpenModel):
226230
locale: str | None = None
227231
message: Any = None

‎astrbot/dashboard/services/stat_service.py‎

Lines changed: 264 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,13 @@
1010
from collections import defaultdict
1111
from datetime import datetime, timedelta, timezone
1212
from functools import cmp_to_key
13+
from html.parser import HTMLParser
1314
from pathlib import Path
15+
from urllib.parse import urljoin, urlparse
1416

1517
import aiohttp
1618
import psutil
19+
from packaging.version import InvalidVersion, Version
1720
from sqlmodel import col, func, select
1821

1922
from astrbot.core import DEMO_MODE, logger
@@ -50,6 +53,22 @@ class StatServiceError(Exception):
5053
pass
5154

5255

56+
# Speed tests download real payloads, so these budgets stop a single click
57+
# from pulling tens of megabytes.
58+
_SPEED_SAMPLE_SECONDS = 6.0
59+
_SPEED_SAMPLE_MAX_BYTES = 4 * 1024 * 1024
60+
# The simple index page of one package lists every release ever published, so
61+
# it can grow without bound on a misbehaving or hostile mirror.
62+
_PACKAGE_PAGE_MAX_BYTES = 8 * 1024 * 1024
63+
_PIP_USER_AGENT = "pip/24.3.1"
64+
# A real repository asset the dashboard downloads (the t2i runtime bundle),
65+
# large enough to produce a meaningful throughput number.
66+
_GITHUB_SPEED_TARGET_URL = (
67+
"https://github.com/AstrBotDevs/AstrBot/raw/refs/heads/master/"
68+
"astrbot/core/utils/t2i/template/shiki_runtime.iife.js"
69+
)
70+
71+
5372
class StatService:
5473
def __init__(
5574
self,
@@ -523,11 +542,12 @@ async def get_provider_token_stats(self, days: int) -> dict:
523542

524543
async def test_ghproxy_connection(self, proxy_url: str | None) -> dict:
525544
try:
526-
if not proxy_url:
527-
raise StatServiceError("proxy_url is required")
528-
529-
proxy_url = proxy_url.rstrip("/")
530-
test_url = f"{proxy_url}/https://github.com/AstrBotDevs/AstrBot/raw/refs/heads/master/.python-version"
545+
target_url = "https://github.com/AstrBotDevs/AstrBot/raw/refs/heads/master/.python-version"
546+
# An empty proxy_url means "no acceleration", so GitHub is tested directly.
547+
normalized_proxy = (proxy_url or "").strip().rstrip("/")
548+
test_url = (
549+
f"{normalized_proxy}/{target_url}" if normalized_proxy else target_url
550+
)
531551
start_time = time.time()
532552

533553
async with (
@@ -550,6 +570,245 @@ async def test_ghproxy_connection(self, proxy_url: str | None) -> dict:
550570
logger.error(traceback.format_exc())
551571
raise StatServiceError(f"Error: {exc!s}") from exc
552572

573+
async def test_pypi_index_connection(self, index_url: str | None) -> dict:
574+
"""Check whether a PyPI index (mirror) answers requests.
575+
576+
Only the response headers are awaited: a simple index page can be very
577+
large, so the body is never downloaded.
578+
579+
Args:
580+
index_url: Base URL of the PyPI index, e.g.
581+
``https://mirrors.aliyun.com/pypi/simple/``.
582+
583+
Returns:
584+
A dict with the round trip latency in milliseconds.
585+
586+
Raises:
587+
StatServiceError: If the URL is missing, malformed, or unreachable.
588+
"""
589+
try:
590+
normalized_url = (index_url or "").strip().rstrip("/")
591+
if not normalized_url:
592+
raise StatServiceError("index_url is required")
593+
594+
parsed = urlparse(normalized_url)
595+
if parsed.scheme not in ("http", "https") or not parsed.netloc:
596+
raise StatServiceError("index_url must be a valid http(s) URL")
597+
598+
start_time = time.time()
599+
async with (
600+
aiohttp.ClientSession() as session,
601+
session.get(
602+
f"{normalized_url}/",
603+
# Some mirrors reject non-pip user agents, so mimic pip itself.
604+
headers={"User-Agent": _PIP_USER_AGENT},
605+
timeout=aiohttp.ClientTimeout(total=10),
606+
) as response,
607+
):
608+
if response.status == 200:
609+
end_time = time.time()
610+
return {
611+
"latency": round((end_time - start_time) * 1000, 2),
612+
}
613+
raise StatServiceError(f"Failed. Status code: {response.status}")
614+
except StatServiceError:
615+
raise
616+
except Exception as exc:
617+
logger.error(traceback.format_exc())
618+
raise StatServiceError(f"Error: {exc!s}") from exc
619+
620+
async def measure_pypi_index_speed(self, index_url: str | None) -> dict:
621+
"""Download a real wheel from a PyPI index and report the throughput.
622+
623+
The speed test pulls an actual package file, so the result reflects what
624+
``pip install`` would experience instead of a bare round trip. Only the
625+
first few megabytes and seconds are sampled so slow mirrors answer fast.
626+
627+
The package listing page is streamed with a hard byte budget, so a
628+
misbehaving mirror cannot push an unbounded page into memory.
629+
630+
Args:
631+
index_url: Base URL of the PyPI index, e.g.
632+
``https://mirrors.aliyun.com/pypi/simple/``.
633+
634+
Returns:
635+
A dict with the throughput in MB/s, the sampled byte count, and the
636+
elapsed seconds.
637+
638+
Raises:
639+
StatServiceError: If the URL is missing, malformed, or unreachable.
640+
"""
641+
try:
642+
normalized_url = (index_url or "").strip().rstrip("/")
643+
if not normalized_url:
644+
raise StatServiceError("index_url is required")
645+
646+
parsed = urlparse(normalized_url)
647+
if parsed.scheme not in ("http", "https") or not parsed.netloc:
648+
raise StatServiceError("index_url must be a valid http(s) URL")
649+
650+
async with aiohttp.ClientSession() as session:
651+
# pip ships a single platform independent wheel and every mirror
652+
# serves the same file, which keeps the numbers comparable.
653+
package_page_url = f"{normalized_url}/pip/"
654+
async with session.get(
655+
package_page_url,
656+
headers={"User-Agent": _PIP_USER_AGENT},
657+
timeout=aiohttp.ClientTimeout(total=15),
658+
) as response:
659+
if response.status != 200:
660+
raise StatServiceError(
661+
f"Failed. Status code: {response.status}"
662+
)
663+
# Stream the page and reject it as soon as it exceeds the
664+
# budget, so an oversized or endless response cannot be
665+
# buffered into memory first.
666+
page_bytes = bytearray()
667+
async for chunk in response.content.iter_chunked(65536):
668+
page_bytes += chunk
669+
if len(page_bytes) > _PACKAGE_PAGE_MAX_BYTES:
670+
raise StatServiceError(
671+
"Package page from this index is too large "
672+
f"(over {_PACKAGE_PAGE_MAX_BYTES // 1048576} MB)"
673+
)
674+
page = page_bytes.decode(
675+
response.charset or "utf-8", errors="replace"
676+
)
677+
678+
class _AnchorHrefParser(HTMLParser):
679+
"""Collect ``href`` values of anchor tags from a simple index.
680+
681+
A real parser is used instead of a regex so that valid markup
682+
with single quotes, unquoted values, or uppercase ``HREF``
683+
is handled; tag and attribute names arrive lowercased.
684+
"""
685+
686+
def __init__(self) -> None:
687+
super().__init__(convert_charrefs=True)
688+
self.hrefs: list[str] = []
689+
690+
def handle_starttag(
691+
self, tag: str, attrs: list[tuple[str, str | None]]
692+
) -> None:
693+
if tag != "a":
694+
return
695+
for name, value in attrs:
696+
if name == "href" and value:
697+
self.hrefs.append(value)
698+
699+
anchor_parser = _AnchorHrefParser()
700+
anchor_parser.feed(page)
701+
anchor_parser.close()
702+
703+
wheels: list[tuple[Version, str]] = []
704+
for href in anchor_parser.hrefs:
705+
# Simple indexes append a "#sha256=..." fragment.
706+
href = href.split("#", 1)[0]
707+
file_name = href.rsplit("/", 1)[-1]
708+
suffix = "-py3-none-any.whl"
709+
if not (
710+
file_name.startswith("pip-") and file_name.endswith(suffix)
711+
):
712+
continue
713+
try:
714+
version = Version(file_name[len("pip-") : -len(suffix)])
715+
except InvalidVersion:
716+
continue
717+
wheels.append((version, urljoin(package_page_url, href)))
718+
719+
if not wheels:
720+
raise StatServiceError("No pip wheel found on this index")
721+
wheels.sort(key=lambda wheel: wheel[0])
722+
wheel_url = wheels[-1][1]
723+
724+
start_time = time.time()
725+
received = 0
726+
async with session.get(
727+
wheel_url,
728+
headers={"User-Agent": _PIP_USER_AGENT},
729+
timeout=aiohttp.ClientTimeout(total=_SPEED_SAMPLE_SECONDS + 5),
730+
) as response:
731+
if response.status != 200:
732+
raise StatServiceError(
733+
f"Failed. Status code: {response.status}"
734+
)
735+
async for chunk in response.content.iter_chunked(65536):
736+
received += len(chunk)
737+
if (
738+
received >= _SPEED_SAMPLE_MAX_BYTES
739+
or time.time() - start_time > _SPEED_SAMPLE_SECONDS
740+
):
741+
break
742+
743+
duration = time.time() - start_time
744+
if not received or duration <= 0:
745+
raise StatServiceError("No data received")
746+
return {
747+
"speed": round(received / 1048576 / duration, 2),
748+
"bytes": received,
749+
"duration": round(duration, 2),
750+
}
751+
except StatServiceError:
752+
raise
753+
except Exception as exc:
754+
logger.error(traceback.format_exc())
755+
raise StatServiceError(f"Error: {exc!s}") from exc
756+
757+
async def measure_ghproxy_speed(self, proxy_url: str | None) -> dict:
758+
"""Download a repository file through a GitHub proxy and report throughput.
759+
760+
Args:
761+
proxy_url: Proxy prefix such as ``https://gh-proxy.com``; an empty
762+
value measures the direct route.
763+
764+
Returns:
765+
A dict with the throughput in MB/s, the sampled byte count, and the
766+
elapsed seconds.
767+
768+
Raises:
769+
StatServiceError: If the download fails.
770+
"""
771+
try:
772+
normalized_proxy = (proxy_url or "").strip().rstrip("/")
773+
test_url = (
774+
f"{normalized_proxy}/{_GITHUB_SPEED_TARGET_URL}"
775+
if normalized_proxy
776+
else _GITHUB_SPEED_TARGET_URL
777+
)
778+
779+
start_time = time.time()
780+
received = 0
781+
async with (
782+
aiohttp.ClientSession() as session,
783+
session.get(
784+
test_url,
785+
timeout=aiohttp.ClientTimeout(total=_SPEED_SAMPLE_SECONDS + 5),
786+
) as response,
787+
):
788+
if response.status != 200:
789+
raise StatServiceError(f"Failed. Status code: {response.status}")
790+
async for chunk in response.content.iter_chunked(65536):
791+
received += len(chunk)
792+
if (
793+
received >= _SPEED_SAMPLE_MAX_BYTES
794+
or time.time() - start_time > _SPEED_SAMPLE_SECONDS
795+
):
796+
break
797+
798+
duration = time.time() - start_time
799+
if not received or duration <= 0:
800+
raise StatServiceError("No data received")
801+
return {
802+
"speed": round(received / 1048576 / duration, 2),
803+
"bytes": received,
804+
"duration": round(duration, 2),
805+
}
806+
except StatServiceError:
807+
raise
808+
except Exception as exc:
809+
logger.error(traceback.format_exc())
810+
raise StatServiceError(f"Error: {exc!s}") from exc
811+
553812
def get_changelog(self, version: str | None) -> dict:
554813
try:
555814
if not version:

‎dashboard/src/api/generated/openapi-v1/sdk.gen.ts‎

Lines changed: 31 additions & 1 deletion
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)