A Streamlit decision-support dashboard that answers one question: given current subsea cable health, which cloud regions/providers are at elevated risk of degraded performance or isolation — and what should you do about it?
Demo.mp4
Subsea cables carry most intercontinental internet traffic. A small number of geographic chokepoints — the Red Sea/Bab-el-Mandeb, the Baltic, Taiwan/Luzon, and Malacca/Singapore — concentrate the risk. A fault at one of these corridors, whether from anchor drag, gray-zone sabotage, or severe weather, can degrade or isolate the cloud regions that depend on it. The global repair fleet is small (~60 specialized ships), and repair SLAs run 2–4 weeks once a fault is confirmed.
No single free, real-time feed connects "cable X is degraded" to "cloud region Y is at risk." This project fuses cable incident data, marine weather over cable corridors, conflict/sabotage news, and cloud provider status into one composite risk view, then uses an LLM to turn that fused state into a structured risk brief — all wrapped in a glassmorphic "Deep Ocean & Electric Cyan" command-center UI.
SIGNAL LAYERS
• Cable incidents (demo injection)
• Marine weather — Open-Meteo (keyless)
• Conflict / sabotage news — Google News RSS (keyless)
• Cloud status probes — AWS / Azure / GCP
• AIS-style vessel correlation (shadow-fleet flagging)
│
▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ INGESTION │─▶ │ VALIDATION │─▶ │ PROCESSING │─▶ │ STORAGE │─▶ │ ENRICHMENT │─▶ │ PRESENTATION │
└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
| Stage | What happens |
|---|---|
| 1. Ingestion | httpx + tenacity async fetch with exponential backoff; feedparser for RSS; degrade-by-design |
| 2. Validation | Pydantic v2 edge models; malformed/schema-mismatched records → quarantine |
| 3. Processing | Polars (LazyFrame, zero-copy) transforms; Pandera quality gates |
| 4. Storage | DuckDB with native GEOMETRY/JSON, spatial + h3 extensions; composite cable risk scoring, H3 aggregation |
| 5. Enrichment | Gemini (google-genai, gemini-3.5-flash) generates a structured risk brief via strict Pydantic JSON schema; full raw-response audit trail |
| 6. Presentation | Streamlit (multipage) + glassmorphic theme; Folium/Leaflet map on Carto dark raster tiles |
- Multi-signal risk fusion — combines cable incident telemetry, marine weather, conflict/sabotage news, and cloud status into one composite risk view.
- Marine weather intelligence — wave, gust, and wind data over cable corridors via Open-Meteo (free, no key) to estimate fault probability and repair-vessel delay.
- Conflict / anchor / sabotage watch — Google News RSS keyword monitoring (free, no key), tagged by zone and severity.
- Cloud status probing — checks public AWS/Azure/GCP status endpoints; mismatched payloads are quarantined instead of crashing the pipeline.
- AIS-style vessel correlation — shadow-fleet flagging built into the incident schema.
- LLM risk briefs — Gemini (
gemini-3.5-flashviagoogle-genai) generates structured, schema-validated risk briefs with a full raw-response audit trail. - Cable status map — Folium/Leaflet map with neon status-colored cable arcs, fault markers, a risk heat layer, and cloud-region nodes.
- Glassmorphic command-center UI — a "Deep Ocean & Electric Cyan" theme with frosted-glass panels and a living header (pulsing LED strip, scrolling ticker, breathing LIVE badge, scanline sweep); pure CSS, no extra JS dependencies.
- Resilient ingestion — async fetch with retry/backoff; malformed data is quarantined, not dropped.
| Category | Technology | Purpose |
|---|---|---|
| App framework | Streamlit (multipage) | Home / Input / Status / Digest pages |
| Data processing | Polars (LazyFrame, zero-copy) | Transforms ingested signals |
| Storage / query | DuckDB (native GEOMETRY + JSON, spatial + h3 extensions) |
Spatial storage, H3 aggregation, composite risk scoring |
| Validation | Pydantic v2 + pydantic-settings | Edge model validation, typed settings |
| Quality gates | Pandera (Polars) | Schema/quality enforcement on processed frames |
| LLM | google-genai (gemini-3.5-flash, free tier) |
Structured JSON risk brief generation |
| Mapping | Folium + Leaflet, Carto dark raster tiles | Token-free interactive map |
| Networking | httpx + tenacity | Async ingestion with exponential backoff |
| Feeds | feedparser | Google News RSS parsing |
| UI theme | Pure CSS (src/theme.py) |
Glassmorphic styling + living effects |
| Logging | loguru | Structured JSON logging |
| Language | Python 3.10+ | Core runtime |
subsea-infrastructure-and-cloud-risk-engine/
├── .github/
│ └── workflows/
│ └── ci.yml # GitHub Actions continuous integration
├── app.py # Home — system status + intelligence basis (glass shell)
├── pages/
│ ├── 1_Input.py # Feed config, live feeds, demo injection, quarantine viewer
│ ├── 2_Status.py # Folium dark map, risk metrics, external-signals panel
│ └── 3_Digest.py # Gemini risk brief generation + audit trail
├── src/
│ ├── __init__.py # Package marker
│ ├── config.py # App configuration / settings
│ ├── schemas.py # Pydantic edge models
│ ├── data_engine.py # Polars processing + Pandera quality gates
│ ├── db_engine.py # DuckDB storage / spatial + H3 logic
│ ├── ingestion.py # Async resilient ingestion (httpx + tenacity)
│ ├── free_feeds.py # Open-Meteo + Google News RSS integrations
│ ├── llm_engine.py # Gemini structured risk brief generation
│ ├── theme.py # "Deep Ocean & Electric Cyan" glass theme + living UI
│ └── viz_layers.py # Folium/Leaflet neon map layer construction
├── tests/ # Pytest suite (37 automated tests)
├── ddl.sql # DuckDB schema definitions
├── requirements.txt # Python dependencies
├── requirements-dev.txt # Dev dependencies (pytest)
├── pyproject.toml # Project metadata / build config
├── pytest.ini # Pytest configuration
└── README.md
git clone https://github.com/Ali-datasmith/subsea-infrastructure-and-cloud-risk-engine.git
cd subsea-infrastructure-and-cloud-risk-engine
pip install -r requirements.txt
# required
export GOOGLE_API_KEY="your-api-key-here"
# optional (default shown)
export GEMINI_MODEL="gemini-3.5-flash"
streamlit run app.py- Push the repo to GitHub and connect it in Streamlit Community Cloud.
- Set Main file path to
app.py. - Set Python version to
3.12. - In Advanced Settings → Secrets, add:
GOOGLE_API_KEY = "your-api-key-here"
GEMINI_MODEL = "gemini-3.5-flash"| Variable | Required | Default | Description |
|---|---|---|---|
GOOGLE_API_KEY |
Yes | — | API key for the google-genai SDK, used for Gemini risk brief generation |
GEMINI_MODEL |
No | gemini-3.5-flash |
Gemini model identifier used for structured-output generation |
No other API keys are required. The weather and news signal layers use free, keyless public endpoints.
| Page | File | Description |
|---|---|---|
| Home | app.py |
System status metrics, the current intelligence basis, and the glass command-center shell |
| Input | pages/1_Input.py |
Feed configuration, live keyless feeds (Open-Meteo weather + Google News RSS), demo incident injection, quarantine viewer, pipeline observability |
| Status | pages/2_Status.py |
Folium dark map with neon status-colored cable arcs, fault markers, risk heat layer, cloud-region nodes; risk summary metrics; external-signals (weather / news / composite score) panel |
| Digest | pages/3_Digest.py |
Gemini risk brief generation, risk brief history, LLM audit trail |
The engine fuses the following signal layers into a composite cable/region risk score:
- Cable fault/incident telemetry — structured incident records. No free real-time subsea fault API exists, so this layer runs on demo injection.
- Marine weather over cable corridors — Open-Meteo wave, gust, and wind data, used as a proxy for fault probability and repair-vessel delay.
- Conflict / anchor / sabotage news — Google News RSS keyword watch, tagged by zone and severity.
- Cloud provider status/latency probes — public AWS/Azure/GCP status endpoints; mismatched payload shapes are quarantined rather than misinterpreted.
- AIS-style vessel correlation — a shadow-fleet flagging model embedded in the incident schema, supporting anchor-drag/sabotage correlation.
- LLM enrichment — Gemini turns the fused, stored state into a structured risk brief via a strict Pydantic schema.
All signals are stored in DuckDB with native GEOMETRY/JSON support and H3-based spatial aggregation, producing a composite cable risk score per corridor.
| Source | Signal | Auth |
|---|---|---|
| Open-Meteo | Marine/weather data over cable corridors | None (keyless) |
| Google News RSS | Conflict/anchor/sabotage keyword monitoring | None (keyless) |
| AWS / Azure / GCP public status endpoints | Cloud provider status probes | None (public) |
Gemini (google-genai) |
Structured risk brief generation | GOOGLE_API_KEY (free tier) |
- Async, resilient ingestion —
httpx+tenacitywith exponential backoff for all external calls. - Degrade-by-design — individual feed failures don't take down the pipeline.
- Quarantine, not crash — malformed or schema-mismatched payloads (e.g., cloud status responses that don't match the expected shape) are routed to a quarantine path, surfaced in the Input page's quarantine viewer, instead of causing failures.
- Edge validation — Pydantic v2 models validate every incoming record at the ingestion boundary.
- Processing-layer gates — Pandera enforces data quality constraints on Polars frames before they reach storage.
- Structured LLM output — Gemini calls use a strict Pydantic JSON schema (
response_mime_type="application/json") instead of freeform text parsing.
- Structured JSON logging via
loguruthroughout ingestion and processing. - Pipeline observability panel on the Input page, surfacing feed health and quarantine activity.
- LLM audit trail on the Digest page — every Gemini request/response is retained for review, alongside risk brief history.
The project includes a comprehensive automated test suite to ensure data integrity, pipeline resilience, and LLM contract compliance. All tests run automatically via GitHub Actions on every push to main.
37 passing tests across 6 core modules:
test_schemas.py: Pydantic edge validation, strict ingestion contracts, and LLM output schemas.test_data_engine.py: Polars LazyFrame transforms, Pandera quality gates, and quarantine logic.test_db_engine.py: DuckDB spatial joins, H3 aggregation, composite risk scoring math, and idempotent upserts (proven viaDELETE + INSERTpattern).test_ingestion.py: Asynchttpx+tenacityretry logic, degrade-by-design orchestration, and malformed payload quarantining.test_free_feeds.py: Open-Meteo marine scoring, Google News RSS parsing, and severity escalation logic.test_llm_engine.py: Gemini structured-output contract validation and JSON fallback parsing.
Run locally:
pip install -r requirements.txt -r requirements-dev.txt
pytest -vThis is a decision-support prototype built entirely on free-tier and public data.
- No free real-time subsea cable fault API exists. The structured cable-fault feed is demo/synthetic (via demo incident injection), not a live production feed.
- Cloud status probes are not latency feeds. AWS/Azure/GCP public status endpoints return incident-shaped data; payloads that don't match the expected shape are quarantined rather than used as latency signal.
- The map basemap is a reliability trade-off. Carto dark raster tiles are used specifically for reliable rendering inside Streamlit Community Cloud's iframe.
Author: Ali-datasmith
See the repository for license details.