mcp-server.mdx

title	MCP Server Reference
description	Complete protocol reference for the html2rss Model Context Protocol (MCP) server, tools, resources, prompts, and transports.

import { Code } from "@astrojs/starlight/components";

html2rss includes a built-in Model Context Protocol (MCP) server that exposes gem capabilities directly to AI agents, clients, and IDEs (such as Cursor, Claude Desktop, and GitHub Copilot).

Overview

The MCP server allows AI agents to inspect pages, capture reusable feed configurations, validate configs, and generate feeds without executing ad-hoc shell scripts or parsing CLI output.

<Code code={`

Start with standard I/O transport (default for AI desktop clients)

html2rss mcp

Start with local HTTP transport (loopback only)

html2rss mcp --transport http --port 8080 `} lang="bash" />

Transports & Security

The MCP server supports two transport mechanisms:

Transport	Command	Default Port	Description
stdio	html2rss mcp --transport stdio	N/A	Standard input/output communication. Recommended for Cursor and Claude Desktop.
http	html2rss mcp --transport http --port 8080	8080	Streamable HTTP transport bound strictly to 127.0.0.1 (loopback only).

Security & Dependencies for HTTP

• Loopback binding: The HTTP transport only binds to 127.0.0.1. Never expose it to public interfaces without authentication and reverse proxy origin controls.
• Dependencies: The HTTP transport requires the rack, rackup, and webrick gems. If they are not present, starting the server raises a descriptive LoadError.
• Logging: stdio JSON-RPC uses stdout. The daemon logs to stderr and defaults to LOG_LEVEL=info unless LOG_LEVEL is set (debug for more, warn to quiet).

Strategy Resolution in MCP

scrape_url and capture_config with strategy: "auto" (the default) run Faraday → Botasaurus AutoFallback in one call.

inspect_url with auto uses Faraday for a cheap diagnostic. Pin strategy: "botasaurus" when you need browser rendering for inspect.

Read html2rss://runtime for botasaurus_configured (boolean only). Set BOTASAURUS_SCRAPER_URL on the MCP process env (see AI agent workflows).

Tool-call budget: scrape_url = 1 call. Durable config = capture_config → validate_config → apply_config. Call inspect_url only when scrape/capture is weak or you need recon. Follow envelope next_step / guidance rather than guessing the next tool.

Result envelope

Every tool returns the same JSON object in both the text content and structuredContent:

<Code code={{ "ok": true, "next_step": "validate_config", "guidance": "...", "payload": {} }} lang="json" />

isError matches ok: false. Scrape results expose articles in payload.items.

next_step is one of: done, inspect_url, validate_config, apply_config, scrape_url, capture_config, read_runtime.

Tools Reference

The server registers 5 primary tools for AI agent orchestration.

scrape_url

One-shot article extraction as JSON Feed items. Use when you need articles immediately without authoring or saving a YAML feed config.

• Parameters:
  • url (string, required): Source page URL to scrape.
  • strategy (string, optional): Request strategy (auto, faraday, botasaurus). Default: auto.
  • limit (integer, optional): Maximum articles to keep. Default: 25.
  • items_selector (string, optional): CSS selector hint for items.
• Return value: Envelope with payload.items (JSON Feed article objects: title, url, description, published_at, author, image, categories). Also payload.total, requested_strategy, channel_title, and admission_drops when present.
• Error handling: Unhandled exceptions return isError: true with payload.class / payload.message. An empty item list is still success (ok: true; follow next_step, which may be inspect_url or read_runtime).

inspect_url

Diagnostic page analysis plus recon from the same fetch. auto uses Faraday; pin botasaurus when you need browser rendering.

• Parameters:
  • url (string, required): Source page URL.
  • strategy (string, optional): Request strategy (auto, faraday, botasaurus). Default: auto.
• Return value: Envelope whose payload is the diagnostic object:
  • requested_url / final_url: requested URL vs Response#url after redirects.
  • status: HTTP status when the strategy reports one.
  • scheme_downgrade: true when the fetch went https → http.
  • alternate_feeds: rel=alternate RSS/Atom links from the document head (no /feed path guessing).
  • content_type / html_response / scraper_eligibility / sst_stats / sst.segment_stats.
  • articles_count and admission_drops when the response is HTML (Cleanup drop-reason counts).
  • blocked_surface when an interstitial signature is detected.
  • xhr_capture when the inspect strategy is botasaurus (count, redacted sample_endpoints, candidate_articles).
• next_step: done when native alternate feeds exist, capture_config when articles were found, otherwise scrape_url.

capture_config

Analyzes a target URL and derives a reusable YAML draft (items selector + enhance: true). Same serializer as CLI html2rss capture. Full schema options live in resource html2rss://schema.

• Parameters:
  • url (string, required): Source page URL.
  • strategy (string, optional): Request strategy (auto, faraday, botasaurus). Default: auto.
  • items_selector (string, optional): Optional CSS selector hint for items.
• Return value: Envelope with payload.yaml (string keys; same serializer as CLI html2rss capture). Treat as a draft: catalog feeds still need directory.topics and explicit channel title/url. Strive to keep enhance: true.
• payload: Also articles_count, channel_title, has_selectors, requested_strategy, and when present segment_strategy, selected_strategy, admission_drops. Follow next_step (validate_config or inspect_url).

validate_config

Validates a feed configuration against the official html2rss JSON schema (html2rss://schema) and runtime constraints.

• Parameters (exactly one of config or yaml):
  • config (object, optional): Feed configuration object.
  • yaml (string, optional): Feed configuration YAML (same form as CLI capture / catalog files).
• Return value: Envelope with empty payload on success (next_step: apply_config).
• Error handling: Returns isError: true with payload.errors if schema validation fails, or next_step: validate_config if both/neither argument is provided.

apply_config

Executes a validated feed configuration against a page and returns RSS 2.0 XML. This is the ship gate: isError when the feed has zero items.

• Parameters:
  • url (string, required): Source page URL (populates channel.url if omitted from config).
  • config (object, optional): Feed configuration object (XOR yaml).
  • yaml (string, optional): Feed configuration YAML (XOR config).
• Return value: Envelope with payload.rss (RSS 2.0 XML) and payload.item_count from the rendered RSS items (not a regex over the XML).
• Error handling: isError: true when the scrape produced no items (payload.item_count: 0, next_step: inspect_url), or on unhandled exceptions.

Resources Reference

The MCP server exposes reference data under the html2rss:// URI scheme:

URI	MIME Type	Description
html2rss://schema	application/json	The complete JSON Schema for html2rss feed configurations, including selector rules, extractors, and request controls.
html2rss://extractors	application/json	Alphabetical list of all registered extractor names (attribute, html, href, text, static, etc.).
html2rss://strategies	application/json	Published MCP request strategies (auto, faraday, botasaurus). Does not list local_file.
html2rss://runtime	application/json	Boolean botasaurus_configured from process env. Never includes the scraper URL.

validate_config / apply_config reject strategy: local_file and request.local_file_path. Use CLI --input for fixtures.

Prompts Reference

The server publishes guided prompt workflows for AI assistants:

scrape-webpage

One scrape_url call with strategy: auto (AutoFallback already hops). Call inspect_url only if the result is weak or you need recon.

• Arguments:
  • url (string, required): Target URL to scrape.

capture-feed-config

Four-step workflow for a durable feed config:

1. Call capture_config — YAML is payload.yaml. Check payload.articles_count and has_selectors. Strive enhance: true.
2. If weak or you need recon, call inspect_url (auto uses Faraday; pin botasaurus for browser rendering).
3. Call validate_config with yaml (or config) — must not be isError.
4. Call apply_config — isError if zero items. Confirm payload.item_count before shipping.

If the destination is html2rss-configs, rewrite the draft with registry.id, directory.topics (non-empty), directory.title, and mirror channel.title (required for Feed Directory configs). Optional directory.summary (max 160 characters). Set explicit channel url.

• Arguments:
  • url (string, required): Target URL to analyze.