Skip to content

ADDING_A_SOURCE.md

Latest commit

 

History

History
247 lines (190 loc) · 10.1 KB

ADDING_A_SOURCE.md

File metadata and controls

247 lines (190 loc) · 10.1 KB
Raw

Adding a marketplace

The shared runtime handles transport, errors, caching and parsing helpers, so a new connector is mostly fetch-and-parse logic. Budget most of your time for step 1 anyway — it decides whether the rest is worth doing.

1. Probe before you build

Anti-bot posture, not API quality, determines feasibility. Megamarket has the cleanest API of any source evaluated for this project and is unusable; Wildberries has a messy one and works perfectly.

Answer these five questions with real HTTP responses, not assumptions:

  1. Does anything answer anonymously? Try the internal endpoints the web client uses, with a realistic Chrome User-Agent and Accept-Language: ru-RU.
  2. What does the block look like? A 403 is different from a redirect loop is different from a JS proof-of-work. Loops (?...&__rr=1) mean IP reputation; proof-of-work means you need a real browser.
  3. Are all three data families reachable? Search, product detail, reviews. A source with detail but no search cannot be discovered through. Lamoda is the case to study: its GraphQL endpoint answers card queries over plain HTTPS but offers no text search, so search had to go through the CDP tier. That split is why the connector carries two transports instead of one.
  4. Does the "search" actually search? Send a distinctive query and read the results. Detsky Mir's API accepts text filters and ignores them, returning its entire 300k catalog. See ANTI_BOT.md.
  5. What happens under load? Five to ten rapid requests. Note the rate limit and whether a captcha appears.

Record the answers, including the negatives. A precise "this returns 403 even with full browser headers" is worth more than an optimistic guess.

2. Decide the transport tier

Tier 1 (anonymous HTTP) if plain HTTPS works. Use mcp_core.transport.http_tier.

Tier 2 (authenticated Chrome) if the source rejects datacenter fingerprints. Use mcp_core.transport.chrome_cdp and add a host allowlist — it runs inside a real session, so a crafted input must never reach a personal endpoint. See CDP_SETUP.md.

Neither if it needs a reversed binary protocol or per-request proof-of-work. Document the refusal in ANTI_BOT.md and stop. Two of six candidates ended here.

3. Scaffold the package

packages/<name>-connector/
├── pyproject.toml
├── src/<name>_connector/
│   ├── __init__.py
│   ├── __main__.py          console-script entry point
│   ├── models_output.py     typed responses
│   ├── settings.py          env-prefixed config
│   ├── server.py            FastMCP tools
│   └── py.typed             PEP 561 marker — do not omit
└── tests/
    ├── conftest.py          needed: test basenames collide across packages
    └── test_server.py

py.typed matters more than it looks: without it, mypy treats every cross-package import as Any and reports phantom errors elsewhere in the tree.

pyproject.toml:

[project]
name = "<name>-connector"
version = "1.0.0"
requires-python = ">=3.12"
dependencies = ["mcp-core", "fastmcp>=3.4.0", "httpx>=0.27", "pydantic>=2.6", "pydantic-settings>=2.2"]

[project.scripts]
<name>-mcp = "<name>_connector.__main__:main"

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/<name>_connector"]

[tool.uv.sources]
mcp-core = { workspace = true }

Then register it in the workspace root pyproject.toml: add to dependencies, [tool.uv.sources], and [tool.pytest.ini_options] testpaths.

4. Write the tools

Four rules carry most of the quality:

Validate inputs by shape, don't escape them. A product id that lands in a URL path is checked with .isdigit(); a category alias is matched against a slug pattern. Rejecting a malformed value is easier to verify than escaping it correctly.

A missing value is None, never 0. Use coerce_price/coerce_int from mcp_core.resilience. A zero price ranks a dead listing as the cheapest option — the exact bug these helpers exist to prevent.

Write field descriptions for a reader who cannot see the API. They are what an LLM uses to decide whether your tool answers the question. Explain semantics, not names: "counts star ratings, not written reviews, which is usually far smaller".

Warn instead of hiding. Partial data, a fallback path, an empty result — all belong in meta.warnings with healthy: false. Silence reads as success.

Every tool needs ToolAnnotations(readOnlyHint=True, destructiveHint=False, idempotentHint=True, openWorldHint=True) and an ## Error Format note in its docstring.

5. Add a tri-state selfcheck

async def <name>_selfcheck(ctx=None) -> SelfcheckResponse:
    # success        — every family answered in the expected shape
    # drift_detected — reachable but unparseable → code change needed
    # inconclusive   — transport/geo/captcha → says nothing about parsers

Chain the probes where you can: have the search probe supply a live id for the card probe, so the canary never depends on a hardcoded SKU that may be delisted.

5a. If you read a rendered page, reuse the shared extractor

Do not write your own DOM helpers. mcp_core.dom exports JS_HELPERS; splice it into your extractor and you inherit four things that were each learned the hard way on a live site:

from mcp_core.dom import JS_HELPERS, prices_from_tile, title_from_tile

_SEARCH_EXTRACT_TEMPLATE = """
() => {
    //__SHARED_HELPERS__
    ...
}
"""
_SEARCH_EXTRACT_JS = _SEARCH_EXTRACT_TEMPLATE.replace("//__SHARED_HELPERS__", JS_HELPERS)
helper what it saves you from
tileRootFor(anchor, idRe) closest() tests the element itself first, so an image link whose class contains "product" becomes the tile — and the tile reads as empty
priceTextsIn(root) splits numbers by whether a currency glyph is attached; a bare number never becomes the price
cleanText / cleanTextWithout textContent, not innerText — the latter depends on layout and is unavailable in the test harness
DECOY_RE instalments, bonuses, discount badges, delivery counts and ratings, generated from one Python list

The extractor must not do arithmetic. Return the display strings and let prices_from_tile decide: coerce_price already understands non-breaking spaces, a missing glyph and comma decimals, and refuses an ambiguous multi-number blob instead of inventing a value.

The one rule to carry: the price is the number attached to a currency glyph. A tile also shows "от 5 751 ₽/ мес.", "- 10%", "в 1356 пунктов" and a rating. A parser that takes the smallest number reports the monthly instalment, and that answer validates, looks right, and is wrong.

6. Test offline

Never let a test touch the network. Monkeypatch the fetch layer and assert the contract an agent sees — which error code, which warning, which fields.

For HTML/SSR sources, capture a real page and trim it rather than inventing markup: preserve the exact nesting so upstream structural changes still surface. The Yandex fixtures went from ~2 MB to ~60 KB this way.

Test the extractor against that markup, not around it. Mocking the render call and feeding a pre-parsed dict leaves the parser itself uncovered — that is exactly where the DNS and Citilink bugs lived while 707 tests stayed green. mcp_core.domtest runs your real extractor over a fixture in jsdom:

from mcp_core.domtest import JsdomUnavailable, run_extractor


def _extract(js_source):
    try:
        return run_extractor(js_source, FIXTURE, page_url="https://example.ru/search/")
    except JsdomUnavailable as exc:
        pytest.skip(str(exc))

jsdom is a developer tool, not a dependency: without it the DOM half skips and the pure-Python price-selection assertions still run. Write both halves.

Record in the fixture header what the page displayed when you captured it — price, old price, availability. That turns the fixture from a blob into evidence someone can re-check.

Mark anything that needs the network @pytest.mark.live and anything needing a browser @pytest.mark.cdp; CI excludes both.

7. Wire it up

  • skills/<name>-connector/SKILL.md — required, and enforced: packages/marketplace-connector/tests/test_skills_parity.py fails the build if a connector has no skill, if the skill omits a tool the server exposes, or if it names a tool that does not exist. Cover when to use it, the workflow patterns, and the gotchas from step 1 — and be explicit about what the source cannot do and which of its answers need a second look. "Run the selfcheck" is not a verification story: a green selfcheck proves the transport answered, not that the parser understood it.
  • compare-connector — add a _search_<name> adapter and an entry in _SEARCH_IMPLS, but only if the source has a working text search. If it does not, leave it out of SEARCHABLE and say why in a comment.
  • CI — bump the expected tool count in .github/workflows/ci.yml and add the console script to the smoke step.
  • README and CHANGELOG — the tool table and the release notes.

8. Verify

uv sync --all-packages
uv run ruff check . && uv run ruff format --check .
uv run mypy
uv run pytest -q -m "not live and not cdp"
uv run python scripts/check_no_print.py

Then run your selfcheck live and confirm the tool inventory:

uv run python -c "
import asyncio
from <name>_connector.server import mcp
print(asyncio.run(mcp.list_tools()))
"

The rule that matters most

If a capability does not exist upstream, do not ship a tool that pretends it does.

Detsky Mir has no text search. A detmir_search tool was written, tested against live data, and deleted — for the query "лего" it returned nappies, dishwashing liquid and a collagen supplement, each with a correct price and rating. That plausibility is what made it dangerous: an error is diagnosable, a confident wrong answer is not.

Document the absence in the tool descriptions so an agent stops looking for a tool that should not exist.