searchstack geo: missing serp_regular function + incorrect AI Overview parser shape

[AO-Raffa]

Hi! Thanks for shipping searchstack — really useful tool. We hit two bugs in searchstack geo while integrating it into our marketing site's SEO/AEO loop. Both reproduce on main (current head) installed via pipx install git+https://github.com/alexpospekhov/searchstack-aeo.git@main.

Reproduction

pipx install git+https://github.com/alexpospekhov/searchstack-aeo.git@main
# Configure .searchstack.toml with [dataforseo] + geo_keywords
searchstack geo "any keyword"
# Output: error: module 'searchstack.providers.dataforseo' has no attribute 'serp_regular'

Bug 1 — dataforseo.serp_regular doesn't exist

commands/geo.py:107 calls dataforseo.serp_regular(...), but providers/dataforseo.py only defines get_auth, api_request, get_location_code, get_language_code. Looks like a half-completed refactor — commands/serp.py builds the same request body inline via api_request.

Bug 2 — _parse_ai_overview reads the wrong response shape

commands/geo.py:13–35:

ai_overview = item.get("ai_overview") or {}
ai_present = bool(ai_overview)

for block in ai_overview.get("items", []):
    for ref in block.get("references", []):
        ...

Per the DataForSEO SERP live advanced docs, AI Overview appears as an item with type="ai_overview" inside task_result["items"], not as a top-level task_result["ai_overview"] field. References live both at the AI Overview item's top level AND nested in items[*].references.

Even with serp_regular added, the current parser will report ai_present=False for every keyword because it's looking in the wrong place.

Bug 3 — request field name

The DataForSEO body key to request AI Overview data is load_async_ai_overview, not load_ai_overview (we initially patched with the wrong name; verified against DataForSEO's how-to-scrape-google-ai-overviews docs). This adds $0.002 per call.

Bug 4 — substring domain matching

Minor: domain in ref_domain or domain in ref_url can false-match (e.g. foo.com matches notfoo.com) and is case-sensitive. Same issue in _parse_organic.

Suggested fixes

Option A — minimal (matches existing pattern)

Inline the request body in geo.py like serp.py does, and rewrite _parse_ai_overview to walk the items array. Most consistent with the rest of the codebase. ~30 lines.

Option B — add a helper

Add serp_regular() to providers/dataforseo.py, then geo.py keeps its current call shape. The patch we're running locally (and have verified produces correct AI Overview data for a Switzerland-targeted query set):

providers/dataforseo.py:

def serp_regular(
    keyword: str,
    location_code: int,
    language_code: str,
    config: Config,
    load_ai_overview: bool = False,
    depth: int = 10,
    device: str = "desktop",
) -> dict[str, Any]:
    """Live Google SERP for a keyword via serp/google/organic/live/advanced.

    load_ai_overview=True maps to the DataForSEO body field `load_async_ai_overview`
    (adds $0.002 per call). The AI Overview appears as an item with type='ai_overview'
    in the result items array.
    """
    body = [{
        "keyword": keyword,
        "location_code": location_code,
        "language_code": language_code,
        "depth": depth,
        "device": device,
    }]
    if load_ai_overview:
        body[0]["load_async_ai_overview"] = True

    data = api_request(config, "serp/google/organic/live/advanced", body)
    if "error" in data:
        raise RuntimeError(data["error"])
    if data.get("status_code") not in (None, 20000):
        raise RuntimeError(f"DataForSEO {data.get('status_code')}: {data.get('status_message', 'unknown error')}")
    tasks = data.get("tasks", [])
    if tasks and tasks[0].get("status_code") not in (None, 20000):
        raise RuntimeError(f"DataForSEO task {tasks[0].get('status_code')}: {tasks[0].get('status_message', 'unknown error')}")
    return data

commands/geo.py _parse_ai_overview:

def _parse_ai_overview(task_result: dict, domain: str) -> dict:
    # Find the ai_overview item among SERP items
    ai_overview_item = None
    for serp_item in task_result.get("items", []) or []:
        if serp_item.get("type") == "ai_overview":
            ai_overview_item = serp_item
            break

    ai_present = ai_overview_item is not None
    cited_domains: list[str] = []
    cites_us = False
    normalized_domain = (domain or "").lower().replace("www.", "")

    def _check_refs(refs):
        nonlocal cites_us
        for ref in refs or []:
            ref_domain = (ref.get("domain") or "").lower().replace("www.", "")
            ref_url = (ref.get("url") or "").lower()
            if ref_domain:
                cited_domains.append(ref_domain)
            if normalized_domain and ref_domain and (
                ref_domain == normalized_domain
                or ref_domain.endswith("." + normalized_domain)
            ):
                cites_us = True
            elif normalized_domain and normalized_domain in ref_url:
                cites_us = True

    if ai_present:
        _check_refs(ai_overview_item.get("references") or [])
        for block in ai_overview_item.get("items") or []:
            _check_refs(block.get("references") or [])

    return {"ai_present": ai_present, "cites_us": cites_us, "cited_domains": cited_domains}

Verified result

With both fixes applied locally, searchstack geo against a 15-keyword set on Switzerland (location_code=2756) returns:

Summary (15 keywords):
  AI Overview present:  7/15
  Cites <our-domain>:   0/15
  Organic top-10:       0/15

Top cited domains in AI Overviews:
  youtube.com (20x)
  cleanbrowsing.org (14x)
  mcafee.com (11x)
  support.google.com (6x)
  unicef.org (6x)
  apps.apple.com (5x)
  internetmatters.org (5x)
  adguard.com (5x)
  nspcc.org.uk (4x)
  kidgy.com (4x)

Happy to open a PR for either option — let me know which you'd prefer. Diagnosis confirmed with a codex (OpenAI) xhigh-reasoning review against the DataForSEO docs.

Thanks for the tool!