Skip to content

README.md

Latest commit

 

History

History
481 lines (374 loc) · 23.9 KB

README.md

File metadata and controls

481 lines (374 loc) · 23.9 KB
Raw

Scripts Directory

This directory contains all deployment and management scripts organized by purpose.

Table of Contents

  • Directory Structure
  • Prerequisites
  • Core Scripts
  • Platform-Specific Scripts
  • Testing Scripts
  • CLI Front Door
  • Profiles Matrix
  • Quick Start Examples
  • Quickstart Matrix
  • Bastion Access and Kubernetes Commands
  • Configuration Precedence
  • Core script flags and dry-run examples
  • Credentials and secrets
  • Ports and local endpoints (defaults)
  • Linting scripts
  • Deprecations
  • Common Env and Flags
  • Troubleshooting playbooks

Directory Structure

  • core/ - Platform-agnostic deployment scripts
  • platform/ - Platform-specific implementations
    • common/ - Shared platform utilities
    • gke/ - Google Kubernetes Engine
    • eks/ - Amazon Elastic Kubernetes Service
    • rke2/ - Rancher Kubernetes Engine
  • aws-proxy/ - AWS reverse proxy management
  • testing/ - Validation and diagnostic scripts
  • troubleshooting/ - Problem resolution scripts

Prerequisites

  • Kubernetes access with kubectl configured for your target cluster
  • Tools: helm (>=3.12), terraform (>=1.5), gcloud, jq, yq, curl
  • For GKE+bastion flows: Terraform workspace at deployment/ applied and a bastion output available (gke_bastion_ssh_command)
  • NVIDIA credentials when deploying NIMs or pulling images: set NGC_API_KEY in .env or export it for the shell session (NGC_API_KEY is accepted as a deprecated alias)
  • Optional: docker for nvcr.io login during env init

Core Scripts

GPU and Infrastructure

  • core/deploy_gpu_operator.sh - Deploys NVIDIA GPU Operator with auto-labeling and conflict resolution
  • core/generate_gpu_configs.sh - Generates GPU-specific configurations

NeMo Deployment

  • core/deploy_nemo.sh - Deploys core NeMo Microservices
  • core/create_nemo_secrets.sh - Creates required NGC and API secrets

Run:AI Integration

  • core/deploy_runai.sh - Deploys Run:AI platform for workload management
    • Supports headless admin bootstrap to auto-create cluster UID/secret.
    • Provide either pre-created cluster credentials (RUNAI_UID/RUNAI_CLIENT_SECRET/RUNAI_TOKEN) or admin bootstrap vars:
      • RUNAI_ADMIN_TOKEN or (RUNAI_ADMIN_CLIENT_ID/RUNAI_ADMIN_CLIENT_SECRET) or (RUNAI_ADMIN_USERNAME/RUNAI_ADMIN_PASSWORD)
      • And a token endpoint: RUNAI_API_TOKEN_URL or RUNAI_AUTH_URL
      • Optional: RUNAI_TENANT_ID, RUNAI_CLUSTER_NAME
    • The script will mint an admin token, create (or find) the cluster via Admin API, fetch install info (clientSecret), persist a k8s Secret runai-cluster-bootstrap, and proceed with Helm install.

Platform-Specific Scripts

CLI Front Door

Use cli.sh for common operations (CLI flags override .env).

Examples (prefer one-command deploys):

# Dev: GPU Operator + NV-Ingest + NIMs (DNS-free)
./scripts/cli.sh deploy --profile dev

# If it seems stuck after "Loading environment...", bypass local detection
./scripts/cli.sh deploy --profile dev --platform gke
# Or explicitly disable detection
./scripts/cli.sh deploy --profile dev --no-detect

# Prod: add domain + TLS
./scripts/cli.sh deploy --profile prod --domain example.com --email admin@example.com

# Full: Run:AI + enterprise add-ons
./scripts/cli.sh deploy --profile full --domain example.com --email admin@example.com

# Optional conveniences
./scripts/cli.sh env init --yes
./scripts/cli.sh status
./scripts/cli.sh diagnose
./scripts/cli.sh destroy --keep-ingress --keep-gpu-operator

Diagnose detection

# Quick check for local kubectl reachability and bastion availability
./scripts/cli.sh diagnose detect

# Check cluster via bastion without local kubeconfig
$(terraform -chdir=$(pwd)/deployment output -raw gke_bastion_ssh_command) --command "timeout 15 kubectl cluster-info"

Profiles Matrix

Profile GPU Operator NV‑Ingest NIMs TLS Run:AI Metrics/Prometheus Enterprise add‑ons
dev Yes Yes Optional No (DNS‑free via nip.io when needed) No Optional via --install-metrics-server, --prometheus No
prod Yes Yes Optional Yes (requires --domain + --email) No Optional via flags No
full Yes Yes Optional Yes if --domain/--email provided; otherwise dev-style Yes Optional via flags Yes (MLFlow, Customizer, LangFlow, HCD)

Notes:

  • NIMs are available via ./scripts/nims deploy ... or as part of local dev flows; bastion full-stack does not enable NIMs by default.
  • CLI flags override .env/env values.

GKE (Google Kubernetes Engine)

Main Deployment

  • platform/gke/deploy_to_bastion.sh - Automated full-stack deployment with multiple modes:
    • --development - Minimal setup for development
    • --production-ready - Production configuration with monitoring
    • --enterprise - Full enterprise features
    • --full-stack - Complete stack with all components
    • --runai - Include Run:AI integration

Component Deployments

  • platform/gke/deploy_nemo_microservices.sh - Complete NeMo suite deployment
  • platform/gke/deploy_nv_ingest.sh - NEW Standalone NV-Ingest deployment
    • Optimized for 20-50x throughput improvement
    • Production-grade resources: 24 CPU cores, 24Gi memory per replica
    • Includes performance tuning for batch processing
    • Auto-scaling configuration
    • GPU node tolerations
  • platform/gke/deploy_nvidia_nims.sh - NVIDIA NIM deployments
  • platform/gke/deploy_llama_nemotron_super.sh - NEW Llama 3.3 Nemotron Super 49B LLM deployment
    • 49B parameter reasoning model (optimized from 70B)
    • Supports reasoning mode (ON/OFF) via system prompt
    • 128K context window
    • Requires 2-4 A100 80GB or 2 H100 80GB GPUs
    • RAG and tool calling capabilities
  • platform/gke/deploy_mlflow.sh - MLFlow for experiment tracking
  • platform/gke/deploy_nemo_customizer.sh - NeMo Customizer with Volcano scheduler
  • platform/gke/deploy_langflow.sh - LangFlow visual AI workflows
  • platform/gke/deploy_hcd.sh - HCD distributed database

Utilities

  • platform/gke/add_missing_components.sh - Add enterprise components post-deployment
  • platform/gke/optimize_nv_ingest_performance.sh - Performance optimization for NV-Ingest
  • platform/gke/patch_nginx_controller_1gb.sh - Canonical NGINX patch tool (controller + ingress profiles)
    • --controller applies global config
    • --ingress all|nv-ingest|embedder|reranker|namespace/name
    • --profile large_upload|high_throughput
    • --whitelist <CIDRs> sets whitelist-source-range
    • --max-body-size <SIZE> sets limit (env default: INGRESS_MAX_BODY_SIZE)
    • --kube-context <CTX> target a specific kube-context
  • platform/gke/monitor_load_balancing.sh - Monitor load distribution across pods
  • platform/gke/monitor_redis_performance.sh - Monitor Redis performance and AOF status (runtime)
  • platform/gke/deploy_ingress.sh - Deploy NGINX Ingress with optional TLS for a domain
  • platform/gke/deploy_ingress_through_bastion.sh - Same as above via bastion
  • platform/gke/setup_bastion_tunnel.sh - Establish on-demand SSH tunnels through bastion

EKS (Amazon Elastic Kubernetes Service)

  • platform/eks/platform_config.sh - EKS-specific configurations

RKE2 (Rancher Kubernetes Engine)

  • platform/rke2/platform_config.sh - RKE2-specific configurations

Testing Scripts

  • testing/validate-deployment.sh - Comprehensive deployment validation
  • testing/diagnose-nemo-deployment.sh - NeMo-specific diagnostics

Quick Start Examples

Advanced (optional)

Most users should not need the patching tools directly. Prefer the one-command deploys above. For advanced tuning:

Enable 1GB File Support (Advanced)

# Apply controller config and ingress large upload profile
./scripts/platform/gke/patch_nginx_controller_1gb.sh --controller --ingress all --profile large_upload

# High throughput profile (adds RPS/connection/keepalive tuning)
./scripts/platform/gke/patch_nginx_controller_1gb.sh --ingress all --profile high_throughput

# Apply IP allowlist (preferred terminology; uses nginx annotation `whitelist-source-range`)
./scripts/cli.sh ingress allowlist --allowed-ips "1.2.3.4/32,5.6.7.0/24"

Deploy NV-Ingest Only

# Basic deployment
./scripts/platform/gke/deploy_nv_ingest.sh --nvidia-api-key "nvapi-xxx"

# Custom namespace and ingress limits
./scripts/platform/gke/deploy_nv_ingest.sh \
  --ngc-api-key "nvapi-xxx" \
  --namespace document-processing \
  --storage-class fast-ssd \
  --max-body-size 3g \
  --whitelist "1.2.3.4/32,5.6.7.0/24"

Full Stack Deployment

# Development setup
./scripts/platform/gke/deploy_to_bastion.sh --development

# Production with custom domain
./scripts/platform/gke/deploy_to_bastion.sh \
  --production-ready \
  --domain mycompany.com \
  --email admin@mycompany.com

Quickstart Matrix

Task Command Prereqs/Notes
Dev deploy ./scripts/cli.sh deploy --profile dev Auto nip.io when needed; optional --local
Prod deploy ./scripts/cli.sh deploy --profile prod --domain <d> --email <e> TLS via cert‑manager; optional --install-metrics-server
Status / Diagnose ./scripts/cli.sh status / ./scripts/cli.sh diagnose Bastion-aware defaults
Port-forward NIMs ./scripts/cli.sh port-forward Opens local 8001/8002/8003
Preflight GPU/GKE ./scripts/cli.sh preflight gke-gpu Checks node OS, conflicts, allocatable GPUs
Fix GKE GPU DS ./scripts/cli.sh fix gke-gpu-conflicts --yes Removes conflicting DS in kube-system
Ingress allowlist ./scripts/cli.sh ingress allowlist --allowed-ips "1.2.3.4/32,5.6.7.0/24" --yes Apply nginx whitelist-source-range annotation

Troubleshooting playbooks

  • NV‑Ingest health: ./scripts/troubleshooting/monitor_nv_ingest_health.sh
  • Validate full stack: ./scripts/cli.sh validate
  • Uneven load/HPA: ./scripts/platform/gke/monitor_load_balancing.sh then ./scripts/cli.sh nginx ingress --profile high_throughput
  • GPU Operator conflicts: ./scripts/cli.sh fix gke-gpu-conflicts --yes, then ./scripts/cli.sh validate

Usage

See the main README.md for detailed deployment instructions.

Notes

  • CLI flags override .env values. Copy env.template to .env as a baseline.
  • Prefer running Kubernetes commands via the bastion resolved from Terraform outputs.

Using scripts/cli.sh

The scripts/cli.sh wrapper provides a simple entrypoint to common operations.

./scripts/cli.sh env init --yes
./scripts/cli.sh status
./scripts/cli.sh diagnose
./scripts/cli.sh destroy --keep-ingress --keep-gpu-operator

CLI command reference (scripts/cli.sh)

  • deploy:
    • Flags: --profile dev|prod|full, --domain DOMAIN, --email EMAIL, --port-forward, --local, --platform gke|eks|rke2, --dry-run, --install-metrics-server, --prometheus, --secrets vault|gcp-sm|1password
    • Orchestrates GPU Operator, controller patching, NV‑Ingest, and NIMs; uses bastion by default on GKE, or --local for current kube-context.
  • nv-ingest deploy [flags...]: forwards to platform/gke/deploy_nv_ingest.sh.
  • nims deploy [flags...]: forwards to platform/gke/deploy_nvidia_nims.sh.
  • nginx controller|ingress [...]: forwards to platform/gke/patch_nginx_controller_1gb.sh.
  • ingress allowlist|whitelist --allowed-ips CSV [--yes] [--dry-run]: convenience allowlist wrapper.
  • monitor lb [--watch]: monitor load distribution.
  • port-forward: open local ports for NV‑Ingest and NIMs via bastion.
  • validate [--no-bastion] [--ingress-max-body-size SIZE]: run validator.
  • preflight gke-gpu: quick GPU/GKE checks.
  • fix gke-gpu-conflicts [--yes]: remove conflicting DaemonSets.
  • env init [--yes] [--non-interactive].
  • status, diagnose, destroy [--keep-ingress] [--keep-gpu-operator] [--platform ...].

Bastion Access and Kubernetes Commands

  • Scripts that need to run kubectl on the bastion use a standardized bastion command resolved from Terraform outputs.
  • You can override it by exporting BASTION_CMD or let scripts auto-detect it from deployment outputs.
  • Example manual usage:
    • $(terraform -chdir=${TF_ROOT:-deployment} output -raw gke_bastion_ssh_command) --command "kubectl get nodes"
    • Explicit path example: $(terraform -chdir=/Users/devin.bost/proj/repos/datastax_nvidia_test_matrix/deployment output -raw gke_bastion_ssh_command) --command "kubectl get nodes"
    • Or set BASTION_TF_DIR to your deployment directory and run the same command
    • Direct fallback (explicit project/zone/name): gcloud compute ssh --project <PROJECT_ID> --zone <ZONE> <BASTION_NAME> --command "kubectl get nodes"

Configuration Precedence

  • CLI flags override environment variables and values from .env.
  • Copy env.template to .env to set defaults; then pass flags for one-off overrides.

Core script flags and dry-run examples

  • create_nemo_secrets.sh

    • Flags: --namespace, --dry-run, --help
    • Example:
      • ./scripts/core/create_nemo_secrets.sh --namespace nemo-microservices
      • ./scripts/core/create_nemo_secrets.sh --namespace nemo-microservices --dry-run

  • deploy_gpu_operator.sh

    • Flags: --platform gke|eks|rke2, --values-file FILE, --dry-run, --help
    • Example:
      • ./scripts/core/deploy_gpu_operator.sh --platform gke
      • ./scripts/core/deploy_gpu_operator.sh --platform gke --dry-run

  • update_gpu_operator_time_slicing.sh

    • Flags: --namespace, --platform gke, --dry-run, --help
    • Example:
      • ./scripts/core/update_gpu_operator_time_slicing.sh --namespace gpu-operator
      • ./scripts/core/update_gpu_operator_time_slicing.sh --namespace gpu-operator --dry-run

  • deploy_nemo.sh

    • Flags: --platform gke|eks|rke2, --values-file PATH, --namespace NS, --dry-run, --help
    • Example:
      • ./scripts/core/deploy_nemo.sh --platform gke --namespace nemo-microservices
      • ./scripts/core/deploy_nemo.sh --platform gke --dry-run

  • deploy_runai.sh

    • Flags: --platform gke|eks|rke2, --domain DOMAIN, --version VER, --dry-run, --help
    • Example:
      • ./scripts/core/deploy_runai.sh --platform gke --domain runai.example.com
      • ./scripts/core/deploy_runai.sh --platform eks --dry-run

Linting scripts

Use the lint helper to check formatting and shellcheck warnings:

./scripts/tools/lint.sh

If tools are missing, install them:

brew install shfmt shellcheck

Deprecations

  • deploy_nginx_to_new_environment.sh → use ./scripts/platform/gke/deploy_ingress.sh ...
  • fix_nginx_413_error.sh → use ./scripts/cli.sh nginx ...
  • patch_ingress_ip_whitelist.sh → use ./scripts/cli.sh whitelist ...
  • platform/gke/patch_ingress_high_throughput.sh → use ./scripts/cli.sh nginx ingress --profile high_throughput

Common Env and Flags

  • Env variables (overview):

    • TF_ROOT: Terraform directory (default: deployment)
    • BASTION_TF_DIR: Optional override for bastion Terraform directory
    • BASTION_CMD: Fully resolved bastion SSH command (overrides Terraform lookup)
    • VIA_BASTION: When true (default behavior in CLI), route kubectl/helm via bastion if available
    • PLATFORM: Target platform for certain scripts (gke|eks|rke2)
    • ALLOWED_IPS: Comma-separated CIDRs for ingress whitelisting
    • INGRESS_MAX_BODY_SIZE: Default ingress body size (e.g., 3g) for controller/ingress
    • NV_INGEST_CHART_VERSION: Preferred variable to pin NV‑Ingest chart version (passed as --chart-version)
    • CHART_VERSION: Remote NV‑Ingest script variable name used when --chart-version is passed
    • HELM_CHART_VERSION: GPU Operator chart version override used by bastion deploy (see GPU Operator section)
    • NGC_API_KEY: Required for NVIDIA image pulls and NIMs
    • NGINX_CHART_VERSION, PROM_CHART_VERSION, CERTM_CHART_VERSION: Optional chart pins for ingress‑nginx, kube‑prometheus‑stack, cert‑manager
    • DEBUG=true: Enables shell tracing (set -x) in most scripts
    • DRY_RUN=true: Print actions without applying changes (where supported)
    • YES=true or CI=true: Assume confirmation for mutating operations

  • Flags (where supported):

    • --yes: Skip confirmation prompts
    • --dry-run: Show what would change without applying
    • --max-body-size SIZE: Set ingress body size (overrides env)
    • --whitelist CIDRS: Set whitelist-source-range for ingresses

Environment Variables Reference (expanded)

CLI flags override exported env vars, which override .env values. Only the most relevant variables are listed here with defaults and behavior notes.

  1. Credentials
  • NGC_API_KEY (required): Canonical NVIDIA/NGC key used for Helm repo auth, Docker registry pulls (nvcr.io), NeMo/NIM deployments, and NV‑Ingest. If missing/invalid, image pulls and Helm fetches will fail.
    • Compatibility note: some scripts accept legacy aliases; new flows should set only NGC_API_KEY.
  1. Bastion and Terraform
  • TF_ROOT (default deployment): Directory used to resolve Terraform outputs.
  • BASTION_TF_DIR: Alternative directory to resolve bastion command from Terraform outputs.
  • BASTION_CMD: Fully resolved bastion SSH command; overrides Terraform lookup if set.
  • VIA_BASTION (default true behavior in CLI): When true and a bastion command is available, kubectl/helm are executed on the bastion. Set to false to force local kubeconfig use.
  1. Ingress and Networking
  • INGRESS_MAX_BODY_SIZE (default 3g): Applied to controller and ingresses. Too low may cause HTTP 413 on large uploads. Validator expects body size plus 600s timeouts and buffering off.
  • ALLOWED_IPS (empty by default): CSV of CIDRs applied to whitelist-source-range. Restricts public access when set.
  • CREATE_EMBEDDER_INGRESS / CREATE_RERANKER_INGRESS (default true): Control whether standalone embedder/reranker ingresses are created in dev flows.
  • Chart pins (optional): NGINX_CHART_VERSION, PROM_CHART_VERSION, CERTM_CHART_VERSION to pin controller, monitoring, and cert‑manager.
  1. NV‑Ingest (standalone optimized deployment)
  • Version pins: prefer NV_INGEST_CHART_VERSION (CLI forwards it as --chart-version), which maps to the remote script’s internal CHART_VERSION variable.
  • NV_INGEST_AZURE_H100 (empty by default): Enable H100‑optimized resources/replicas. Use only on Azure NC80adis_H100_v5 clusters.
  • NV_INGEST_ENABLE_CAPTIONING (empty): Enables local VLM image captioning microservice.
  • NV_INGEST_CAPTION_MODEL (default meta/llama-3.2-11b-vision-instruct): Caption model used when captioning is enabled.
  • Custom CA (optional): CA_BUNDLE_CONFIGMAP, CA_BUNDLE_KEY (default ca.crt), CA_MOUNT_PATH (default /etc/ssl/certs/custom-ca).
    • Bastion caveat: when deploying via bastion, these envs are not implicitly forwarded; pass --ca-* flags to deploy_nv_ingest.sh to enforce CA wiring, or update bastion env handoff.
  • Namespaces/Scheduler: NVINGEST_NAMESPACE (default nv-ingest in CLI dev helper), NVINGEST_SCHEDULER (optional default|runai argument in deploy_nv_ingest.sh).
  • Local chart development: LOCAL_CHART_PATH is supported for local installs; not applicable when executing on a remote bastion.
  1. NeMo Microservices
  • Scheduler: NEMO_SCHEDULER (default auto): resolves to runai if Run:AI detected; else volcano if Volcano detected; otherwise defaults to Kubernetes scheduler. INSTALL_VOLCANO (default true) auto‑installs Volcano when needed.
  • Namespace: NEMO_NAMESPACE (default nemo-microservices).
  • NV‑Ingest in NeMo chart: NV_INGEST_HELM_CHART_VERSION and NV_INGEST_IMAGE_TAG apply only to the NV‑Ingest component inside the NeMo Helm chart (distinct from the standalone NV‑Ingest deployment).
  1. GPU Operator
  • GPU_OPERATOR_VERSION (default v25.3.1): Helm chart version used by deploy_gpu_operator.sh.
  • HELM_CHART_VERSION (optional): When set, deploy_to_bastion.sh overrides GPU_OPERATOR_VERSION with this value for the operator. Keep the two in sync if you set both.
  • GPU_OPERATOR_MANAGED_DRIVER (default true): Guardrail for a post‑install check that allocatable GPUs become available; setting it false skips the guard (does not switch driver mode by itself).
  • VALIDATION_STRICT (default true): Causes validators/bastion flow to treat GPU conflicts/allocatable=0 as fatal.
  1. Run:AI (optional)
  • Credentials: RUNAI_TOKEN, RUNAI_CLIENT_SECRET, RUNAI_UID (required for full install unless admin bootstrap is used).
  • Admin/bootstrap: RUNAI_ADMIN_TOKEN or (RUNAI_ADMIN_CLIENT_ID/RUNAI_ADMIN_CLIENT_SECRET) or (RUNAI_ADMIN_USERNAME/RUNAI_ADMIN_PASSWORD) with RUNAI_API_TOKEN_URL or RUNAI_AUTH_URL.
  • Domain/TLS: RUNAI_DOMAIN, RUNAI_CERT_PATH, RUNAI_KEY_PATH (paths must exist on the bastion host when deploying through bastion).
  • Swap/tuning: RUNAI_ENABLE_GPU_SWAP (default false), RUNAI_SWAP_CPU_RAM, RUNAI_SWAP_RESERVED_GPU_RAM, RUNAI_SWAP_NODE_SELECTOR, RUNAI_SCHEDULER_NAME.
    • Interaction: when swap is enabled, the operator time‑slicing configuration is skipped by design.
  1. Misc/Reserved
  • SECRETS_MANAGER (CLI flag/env): Currently a pass‑through hint; default flows create Kubernetes Secrets directly. Future work may wire managers like Vault/GCP‑SM/1Password.
  • LOG_LEVEL (CLI flag): Parsed by the CLI but not applied across sub‑scripts; treated as informational today.
  • KUBECTL_CONTEXT: Parsed by the CLI; not consistently applied by sub‑scripts.

Credentials and secrets

  • Image pulls and NIM deployments use NGC_API_KEY and/or NGC_API_KEY.
  • Run ./scripts/cli.sh env init to create .env and optionally perform nvcr.io login.
  • For Kubernetes secrets used by NeMo and related services, see ./scripts/core/create_nemo_secrets.sh (supports --namespace and --dry-run).
  • The high-level deploy command accepts --secrets to select a secrets manager (k8s|vault|gcp-sm|1password).

Examples:

# Kubernetes (default): create k8s Secrets directly
SECRETS_MANAGER=k8s ./scripts/cli.sh deploy --profile prod --domain <d> --email <e>

# HashiCorp Vault: provide VAULT_*; downstream scripts should materialize k8s Secrets from Vault values
SECRETS_MANAGER=vault VAULT_ADDR=https://vault.example.com VAULT_TOKEN=... ./scripts/cli.sh deploy --profile prod --domain <d> --email <e>

# Google Secret Manager: ensure GCP auth; scripts fetch and create k8s Secrets
SECRETS_MANAGER=gcp-sm GCP_PROJECT_ID=<project> ./scripts/cli.sh deploy --profile prod --domain <d> --email <e>

# 1Password: provide service account token; scripts fetch and create k8s Secrets
SECRETS_MANAGER=1password OP_SERVICE_ACCOUNT_TOKEN=op://... ./scripts/cli.sh deploy --profile prod --domain <d> --email <e>

Notes:

  • Current default flows create Kubernetes Secrets directly (k8s). Other managers are pass-through hints; component scripts are responsible for fetching and templating secret data.
  • scripts/core/create_nemo_secrets.sh creates required k8s Secrets for NeMo.

Ports and local endpoints (defaults)

These are used by ./scripts/setup-nvidia-nims-portforward.sh and can be overridden in .env:

  • NV‑Ingest: NVINGEST_PROD_PORT=7670, NVINGEST_DEV_PORT=7671
  • Embedder (original): EMBEDDER_ORIGINAL_PORT=8001
  • Embedder (v2): EMBEDDER_V2_PORT=8002
  • Reranker: RERANKER_PORT=8003

Quick check after ./scripts/cli.sh port-forward:

curl http://localhost:7670/v1/health/live   # NV‑Ingest prod
curl http://localhost:8001/v1/models        # Embedder (original)

Troubleshooting index

  • GPU Operator conflicts or allocatable GPUs = 0:
    • ./scripts/cli.sh preflight gke-gpu
    • ./scripts/cli.sh fix gke-gpu-conflicts --yes
    • ./scripts/cli.sh validate
  • NV‑Ingest health and uneven load:
    • ./scripts/troubleshooting/monitor_nv_ingest_health.sh
    • ./scripts/platform/gke/monitor_load_balancing.sh --watch
  • Ingress 413 (body too large) or timeouts:
    • Controller baseline: ./scripts/cli.sh nginx controller --yes
    • Large upload/high throughput profiles: ./scripts/cli.sh nginx ingress --target all --profile large_upload|high_throughput --yes
  • Apply IP allowlist:
    • ./scripts/cli.sh ingress allowlist --allowed-ips "1.2.3.4/32,5.6.7.0/24" --yes