feat(serve): describe an endpoint's traffic and size workloads from it (#6552)

Add describe_endpoint_traffic(), which describes the traffic one model on
an endpoint served over a window, relative (lookback back from end_time,
seven days by default) or absolute (start_time to end_time), up to seven
days. Times must be timezone-aware and are reported in UTC.

- Concurrency for the whole deployment: each hour's maximum
  ConcurrentRequestsPerCopy or ConcurrentRequestsPerModel times the
  current copies or instances.
- With detailed observability on, the request count and the prompt and
  generated tokens of the requests the model completed, from the serving
  container's token histograms. They're read through CloudWatch's
  Prometheus-compatible query API with the session's credentials, over the
  CloudWatch client's HTTP session (its proxy, CA bundle and timeouts),
  with retries on throttling, server errors and dropped connections. The
  read is best effort: a failure is reported in token_metrics_error and
  logged. The histograms are listed per serving framework (vLLM today),
  and the result names the framework read. They aren't read when the
  endpoint configuration turns detailed observability off.
- Tokens and concurrency are TrafficStats: min, p25, p50, p70, p80, p90,
  p95, p99, max (p100), mean and standard deviation. Token percentiles
  interpolate within the histograms' buckets. Concurrency's are over the
  window's hourly maximums, so its max is the busiest hour's.
- The variant's Data Capture prefix, when the endpoint captures requests.

The histograms are cumulative, so a series' traffic in the window is the
sum of its increments between daily snapshots, from the start's to the
window's last. A snapshot below the one before it follows a restart of
its container and counts from zero.

An adapter's requests run in its base component's container, which
publishes the concurrency and the token histograms for them, so an
adapter is described by its base component, and finding the component
skips adapters.

Add Workload.from_endpoint(), which builds a synthetic workload from the
token metrics at the busiest hour's concurrency, or with
use_data_capture=True replays the variant's captured requests, so a
benchmark or recommendation job can be sized from what an endpoint
serves today. Other workload parameters, such as tokenizer and hf_token,
pass through to the workload.

Integration tests: a no-GPU check that the signed token query is
accepted and parsed by CloudWatch's query API, and an e2e step that
describes the benchmarked endpoint's traffic and runs a benchmark sized
from it.

Author: ZealSV

sagemaker-serve/src/sagemaker/serve/ai_inference_recommender/__init__.py

@@ -39,6 +39,11 @@
     compare_benchmarks,
 )
 from sagemaker.serve.ai_inference_recommender.secrets import Secret
+from sagemaker.serve.ai_inference_recommender.traffic import (
+    EndpointTraffic,
+    TrafficStats,
+    describe_endpoint_traffic,
+)
 from sagemaker.serve.ai_inference_recommender.workload import Workload
 from sagemaker.serve.ai_inference_recommender._model_builder_methods import (
     start_benchmark,
@@ -51,14 +56,17 @@
     "BenchmarkMetrics",
     "BenchmarkResult",
     "BenchmarkSearchResult",
+    "EndpointTraffic",
     "FeatureGatedError",
     "InferenceFramework",
     "PerformanceTarget",
     "RecommendationJob",
     "Secret",
+    "TrafficStats",
     "Workload",
     "WorkloadValidationError",
     "compare_benchmarks",
+    "describe_endpoint_traffic",
     "list_benchmarks",
     "list_recommendations",
     "start_benchmark",