feat: unify crawl engine, harden scan jobs, and enrich the link graph

Share one crawl engine between /v1/site-audit and persistent link-graph
scans, then make background scans survive process restarts and represent
non-200 pages faithfully in the graph.

Crawl engine
- Extract SiteCrawler.crawl returning a SiteCrawlSnapshot; link_graph now
  consumes it instead of running a second BFS, so robots, sitemap seeding,
  URL normalization, and depth/page budgets behave identically everywhere.
- Add cooperative cancellation and progress reporting to SiteCrawler.
- Skip non-HTML asset URLs at enqueue time via shared is_html_like_url.
- Expose use_sitemap through the scan API and persisted scan options.

Scan jobs
- Bound concurrent scans with SEO_SCAN_JOB_WORKERS and add worker leases,
  heartbeats, and a reconcile loop governed by SEO_SCAN_JOB_LEASE_SECONDS.
- Coordinate cancellation through SQLite so any process can cancel a scan.
- Requeue in-flight work on graceful shutdown; recover abandoned leases.
- Enable WAL and a busy timeout; give in-memory storage a shared-cache DSN.

Link graph
- Model redirects, robots-blocked, fetch-failed, and out-of-scope pages as
  real graph nodes with redirect edges and preserved chains.
- Merge link zones (header/nav/content/footer) per target and carry them
  into edges, incoming links, and dashboard filters.
- Recompute inbound/outbound counts from stats and apply issue penalties
  incrementally with a matching grade.

Interfaces
- Escape inline dashboard JSON against script-tag breakout.
- Add cancel/rerun/dashboard controls and error surfacing to the browser UI;
  fetch the dashboard with the API key instead of a bare link.
- Persist SQLite in a dedicated /data volume for the read-only container.
- Document every link-graph route, scan data retention, and limitations.

Tests: 100 passed, 90% coverage; ruff, mypy, and pip-audit clean.

Author: hlibsuslov

.env.example

@@ -17,6 +17,7 @@ SEO_CACHE_MAX_ENTRIES=512
 SEO_MAX_SITE_PAGES=100
 SEO_SCAN_STORAGE_PATH=data/analyzer.db
 SEO_SCAN_JOB_WORKERS=2
+SEO_SCAN_JOB_LEASE_SECONDS=30
 
 # Optional Google PageSpeed Insights integration. It is off by default because it
 # consumes an external quota and makes analysis slower.

Dockerfile

@@ -13,14 +13,16 @@ FROM python:3.13-slim@sha256:6771159cd4fa5d9bba1258caf0b82e6b73458c694d178ad97c5
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \
     PORT=8000 \
-    SEO_SCAN_STORAGE_PATH=/tmp/analyzer.db
+    SEO_SCAN_STORAGE_PATH=/data/analyzer.db
 
-RUN groupadd --system app && useradd --system --gid app --home-dir /app app
+RUN groupadd --system app && useradd --system --gid app --home-dir /app app \
+    && mkdir -p /data && chown app:app /data
 WORKDIR /app
 COPY --from=builder /wheels /wheels
 RUN python -m pip install --no-cache-dir /wheels/* && rm -rf /wheels
 COPY --chown=app:app main.py ./main.py
 USER app
+VOLUME ["/data"]
 EXPOSE 8000
 HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
   CMD python -c "import os,urllib.request; urllib.request.urlopen('http://127.0.0.1:'+os.getenv('PORT','8000')+'/healthz', timeout=2)"

README.md

@@ -17,6 +17,25 @@ This fork is a ground-up v2 implementation of [KovalDenys1/SEO-Analyzer-API](htt
 - Ranks supplied Search Console/conversion rows by traffic and revenue opportunity without inventing external keyword or SERP data.
 - Optionally enriches a page with Google PageSpeed Insights v5 lab/field data.
 
+## Architecture
+
+```mermaid
+flowchart LR
+    Client[API or browser UI] --> API[FastAPI]
+    API --> Jobs[Bounded scan manager]
+    Jobs --> Crawler[Shared SiteCrawler]
+    Crawler --> Fetcher[SafeFetcher]
+    Crawler --> Parser[HTML parser and SEO scoring]
+    Parser --> Graph[Link graph and site-level issues]
+    Graph --> Storage[(SQLite)]
+    Storage --> API
+    API --> Dashboard[Interactive graph dashboard]
+```
+
+`/v1/site-audit` and persistent link-graph scans use the same crawler engine.
+Each fetched document is parsed once, then reused for SEO scoring, internal and
+external edges, duplicate/orphan/broken-link checks, storage, and visualization.
+
 ## Safety by default
 
 The service fetches user-supplied URLs, so URL handling is part of the security boundary:
@@ -43,8 +62,20 @@ Private-network access can be enabled for a trusted internal deployment, but it
 | `POST` | `/v1/opportunities` | First-party traffic/revenue opportunity ranking |
 | `GET` | `/app` | Minimal browser UI for unified link-graph scans |
 | `POST` | `/api/projects` | Create a persistent crawl project |
+| `GET` | `/api/projects` | List projects |
+| `GET` | `/api/projects/{project_id}` | Get one project |
 | `POST` | `/api/projects/{project_id}/scans` | Start a background crawl + graph + SEO scan |
+| `GET` | `/api/projects/{project_id}/scans` | List project scans |
+| `GET` | `/api/scans/{scan_id}/status` | Get progress and terminal status |
+| `POST` | `/api/scans/{scan_id}/cancel` | Cancel pending or running work |
+| `POST` | `/api/scans/{scan_id}/rerun` | Create a new scan with the same options |
+| `GET` | `/api/scans/{scan_id}/pages` | List page records and attached SEO data |
+| `GET` | `/api/scans/{scan_id}/page?url=...` | Resolve a page by normalized URL |
+| `GET` | `/api/scans/{scan_id}/pages/{graph_node_id}` | Resolve a page by graph node ID |
+| `GET` | `/api/scans/{scan_id}/links` | List internal, external, or redirect links |
 | `GET` | `/api/scans/{scan_id}/graph` | Graph nodes/edges with SEO data attached |
+| `GET` | `/api/scans/{scan_id}/seo/issues` | List page and site-level SEO issues |
+| `GET` | `/api/scans/{scan_id}/stats` | Site totals, duplicates, cycles, orphans, and failures |
 | `GET` | `/api/scans/{scan_id}/dashboard` | Interactive graph dashboard for a completed scan |
 | `GET` | `/analyze` | Backwards-compatible original full-analysis shape plus `v2` data |
 | `GET` | `/quick-score` | Score, warnings, and top recommendations |
@@ -107,7 +138,9 @@ If `SEO_API_KEY` is set, add `-H 'X-API-Key: …'` to protected endpoints.
 
 ## Docker
 
-The image runs as a non-root user. The Compose example binds only to loopback, drops Linux capabilities, uses a read-only filesystem, and adds a health check.
+The image runs as a non-root user. The Compose example binds only to loopback,
+drops Linux capabilities, uses a read-only root filesystem, and persists SQLite
+in the `analyzer-data` volume mounted at `/data`.
 
 ```bash
 cp .env.example .env
@@ -129,7 +162,8 @@ All settings use the `SEO_` prefix. See [`.env.example`](.env.example) for the c
 | `SEO_CACHE_TTL_SECONDS` | `300` | Analysis cache TTL; `0` disables cache |
 | `SEO_MAX_SITE_PAGES` | `100` | Server-side hard cap for a site audit |
 | `SEO_SCAN_STORAGE_PATH` | `data/analyzer.db` | SQLite storage for persistent link-graph scans |
-| `SEO_SCAN_JOB_WORKERS` | `2` | Reserved scan worker budget for deployments |
+| `SEO_SCAN_JOB_WORKERS` | `2` | Concurrent link-graph scans per API process |
+| `SEO_SCAN_JOB_LEASE_SECONDS` | `30` | Time before an abandoned running scan is requeued |
 | `SEO_ENABLE_PAGESPEED` | `false` | Permit quota-consuming PageSpeed calls |
 | `SEO_PAGESPEED_API_KEY` | empty | Optional Google API key |
 | `SEO_CORS_ORIGINS` | empty | Comma-separated browser origins |
@@ -152,6 +186,23 @@ pip-audit
 
 The suite covers URL security, DNS/redirect validation, parsing, page classification, issue scoring, sitemap/robots behavior, crawl aggregation, PageSpeed normalization, API compatibility, auth, and opportunity ranking.
 
+## Scan data
+
+SQLite stores projects, scans, normalized page URLs, stable graph node IDs,
+links with page zones, SEO reports, status/depth/timing metadata, and the complete
+versioned result snapshot. HTML response bodies are not persisted. Running scans
+write a worker lease and heartbeat; graceful shutdown requeues active work, while
+an abandoned lease is recovered automatically.
+
+## Known limitations
+
+- Crawls are bounded samples and do not execute client-side JavaScript.
+- External targets are represented in the graph but are not fetched or checked.
+- SQLite is intended for local and small-team deployments; high-volume distributed
+  deployments should replace the storage/job backend.
+- Worker limits apply per API process. Global fetch concurrency is still bounded
+  by `SEO_MAX_CONCURRENT_FETCHES` in each process.
+
 ## Documentation
 
 - [API guide](docs/API.md)
@@ -163,4 +214,7 @@ The suite covers URL security, DNS/redirect validation, parsing, page classifica
 
 ## License and attribution
 
-MIT. The upstream project is copyright Denys Koval; this fork preserves the original license and history. See [LICENSE](LICENSE).
+MIT. The upstream project is copyright Denys Koval. Link-graph functionality is
+derived from [ruslan2027/link-graph-explorer-oss](https://github.com/ruslan2027/link-graph-explorer-oss).
+See [LICENSE](LICENSE), [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md), and
+[licenses/link-graph-explorer-oss.LICENSE](licenses/link-graph-explorer-oss.LICENSE).

compose.yaml

@@ -6,11 +6,15 @@ services:
     env_file:
       - path: .env
         required: false
+    environment:
+      SEO_SCAN_STORAGE_PATH: /data/analyzer.db
     ports:
       - "127.0.0.1:8000:8000"
     read_only: true
     tmpfs:
       - /tmp:size=32m,mode=1777
+    volumes:
+      - analyzer-data:/data
     security_opt:
       - no-new-privileges:true
     cap_drop:
@@ -19,3 +23,6 @@ services:
       resources:
         limits:
           memory: 512M
+
+volumes:
+  analyzer-data:

docs/LINK_GRAPH_INTEGRATION.md

@@ -9,6 +9,9 @@ adds persistent link-graph scans on top of the existing safe asynchronous
 - The existing `SafeFetcher` remains the only network layer. The graph scan does
   not add a second crawler with `requests`, so SSRF protections, DNS validation,
   redirect validation, response budgets, and concurrency limits remain intact.
+- `/v1/site-audit` and persistent graph scans share `SiteCrawler.crawl`, including
+  robots handling, sitemap discovery, URL normalization, redirect deduplication,
+  and page/depth/concurrency budgets.
 - Each fetched HTML page is analyzed once by `Analyzer.analyze_artifact`. The
   resulting parsed links and SEO report are reused for graph nodes, edges, page
   details, issue lists, stats, and the dashboard.
@@ -23,13 +26,15 @@ adds persistent link-graph scans on top of the existing safe asynchronous
 - `seo_analyzer.link_graph`: async BFS graph scan, page normalization, graph
   stats, cycles, duplicates, redirects, and SEO-to-node mapping.
 - `seo_analyzer.storage`: SQLite persistence.
-- `seo_analyzer.jobs`: background scan tasks, progress, cancellation, and rerun.
+- `seo_analyzer.jobs`: bounded background scans, SQLite worker leases, heartbeat,
+  cross-worker cancellation, automatic recovery, and rerun.
 - `seo_analyzer.dashboard`: completed-scan graph dashboard renderer.
 - `seo_analyzer.frontend`: minimal browser UI for starting scans.
 
 ## Limitations
 
-- Jobs are in-process and are not resumed after an API process restart.
+- Active work is executed in-process. Graceful shutdown immediately requeues it;
+  process crashes are recovered after `SEO_SCAN_JOB_LEASE_SECONDS`.
 - External links are collected and visualized, but external targets are not fully
   crawled.
 - The dashboard is intentionally dependency-free and compact; deeper graph

docs/SECURITY.md

@@ -44,11 +44,21 @@ Analysis responses add a request ID, `nosniff`, a no-referrer policy and `Cache-
 
 PageSpeed is disabled by default. When enabled and requested, the final validated public URL is sent to Google PageSpeed Insights. That creates a third-party data and quota dependency; document it in your privacy/processing model and protect `SEO_PAGESPEED_API_KEY` in the environment.
 
-Page HTML is processed in memory and cached in summarized analysis form for a bounded TTL. This project has no database and does not intentionally persist fetched HTML. Multi-worker deployments have independent caches and metrics.
+Page HTML is processed in memory and cached in summarized analysis form for a
+bounded TTL. Persistent scans store normalized URLs, extracted metadata, links,
+SEO findings, graph data, and status/timing summaries in SQLite; raw HTML response
+bodies are not intentionally persisted. Multi-worker deployments have independent
+caches and metrics, while scan ownership and cancellation are coordinated through
+SQLite worker leases.
 
 ## Container posture
 
-The supplied image runs as a non-root system user. Compose binds to loopback, drops all Linux capabilities, sets `no-new-privileges`, uses a read-only root filesystem, provides a small `/tmp` tmpfs and declares a memory limit. Put a production reverse proxy in front; do not expose the Uvicorn development topology as a complete security perimeter.
+The supplied image runs as a non-root system user. Compose binds to loopback,
+drops all Linux capabilities, sets `no-new-privileges`, uses a read-only root
+filesystem, persists scan data in a dedicated `/data` volume, provides a small
+`/tmp` tmpfs, and declares a memory limit. Put a production reverse proxy in
+front; do not expose the Uvicorn development topology as a complete security
+perimeter.
 
 Container restrictions do not replace outbound firewall rules. For higher-risk deployments, allow egress only to public HTTP(S), run in a dedicated network/namespace, configure DNS deliberately and set infrastructure-level CPU/request/time limits.
 

pyproject.toml

@@ -9,6 +9,7 @@ description = "Security-first SEO and growth-strategy analyzer for SaaS websites
 readme = "README.md"
 requires-python = ">=3.11"
 license = "MIT"
+license-files = ["LICENSE", "THIRD_PARTY_NOTICES.md", "licenses/*.LICENSE"]
 authors = [{ name = "hlibsuslov", email = "glebsuslov720@gmail.com" }]
 keywords = ["seo", "saas", "fastapi", "crawler", "technical-seo"]
 classifiers = [

seo_analyzer/api.py

@@ -109,6 +109,7 @@ class ScanCreate(BaseModel):
     respect_robots: bool = True
     include_subdomains: bool = False
     include_query_parameters: bool = False
+    use_sitemap: bool = True
 
 
 def create_app(settings: Settings | None = None, analyzer: Analyzer | None = None) -> FastAPI:
@@ -129,9 +130,14 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
             app.state.scan_storage,
             app.state.analyzer,
         )
-        yield
-        if owns_analyzer:
-            await app.state.analyzer.close()
+        await app.state.scan_manager.startup()
+        try:
+            yield
+        finally:
+            await app.state.scan_manager.shutdown()
+            app.state.scan_storage.close()
+            if owns_analyzer:
+                await app.state.analyzer.close()
 
     app = FastAPI(
         title="SaaS SEO Analyzer API",
@@ -484,6 +490,7 @@ async def start_scan(
             respect_robots=payload.respect_robots,
             include_subdomains=payload.include_subdomains,
             include_query_parameters=payload.include_query_parameters,
+            use_sitemap=payload.use_sitemap,
         )
         return manager.create_and_start(project_id, start_url, options)
 
@@ -557,6 +564,7 @@ async def scan_pages(
         pages = storage.list_pages(scan_id, include_redirects=include_redirects)
         return _filter_pages(
             pages,
+            include_redirects=include_redirects,
             status=status,
             max_depth=max_depth,
             issue=issue,
@@ -573,8 +581,14 @@ async def scan_page_by_url(
         url: str = Query(min_length=4, max_length=2_048),
         storage: ScanStorage = Depends(get_scan_storage),
     ) -> dict[str, Any]:
-        _require_scan(scan_id, storage)
-        page = storage.get_page_by_url(scan_id, normalize_url(url, keep_query=True))
+        scan = _require_scan(scan_id, storage)
+        page = storage.get_page_by_url(
+            scan_id,
+            normalize_url(
+                url,
+                keep_query=bool(scan["options"].get("include_query_parameters", False)),
+            ),
+        )
         if not page:
             raise HTTPException(status_code=404, detail="Page not found")
         return page
@@ -602,7 +616,7 @@ async def scan_page_by_node(
     )
     async def scan_links(
         scan_id: str,
-        type: str | None = Query(default=None, pattern="^(internal|external)$"),
+        type: str | None = Query(default=None, pattern="^(internal|external|redirect)$"),
         storage: ScanStorage = Depends(get_scan_storage),
     ) -> list[dict[str, Any]]:
         _require_scan(scan_id, storage)
@@ -625,6 +639,7 @@ async def scan_graph(
         if any(value is not None for value in (status, max_depth, issue, min_score)):
             pages = _filter_pages(
                 list(result["crawl"]["pages"].values()),
+                include_redirects=True,
                 status=status,
                 max_depth=max_depth,
                 issue=issue,
@@ -739,18 +754,19 @@ def _require_result(scan_id: str, storage: ScanStorage) -> dict[str, Any]:
 def _filter_pages(
     pages: list[dict[str, Any]],
     *,
+    include_redirects: bool = False,
     status: int | None = None,
     max_depth: int | None = None,
     issue: str | None = None,
     min_score: float | None = None,
 ) -> list[dict[str, Any]]:
     filtered = []
     for page in pages:
-        if page.get("redirected_to"):
+        if page.get("redirected_to") and not include_redirects:
             continue
         if status is not None and int(page.get("status") or 0) != status:
             continue
-        if max_depth is not None and int(page.get("depth") or 0) > max_depth:
+        if max_depth is not None and (page.get("depth") is None or int(page["depth"]) > max_depth):
             continue
         seo = page.get("seo") or {}
         if min_score is not None and float(seo.get("score") or 0) < min_score:

seo_analyzer/config.py

@@ -31,6 +31,7 @@ class Settings(BaseSettings):
     max_site_pages: int = Field(default=100, ge=1, le=1_000)
     scan_storage_path: str = "data/analyzer.db"
     scan_job_workers: int = Field(default=2, ge=1, le=16)
+    scan_job_lease_seconds: int = Field(default=30, ge=10, le=600)
 
     enable_pagespeed: bool = False
     pagespeed_api_key: SecretStr | None = None