Skip to content

cli-reference.mdx

Latest commit

 

History

History
234 lines (166 loc) · 8.82 KB

cli-reference.mdx

File metadata and controls

234 lines (166 loc) · 8.82 KB
Raw
title CLI Reference
description Complete reference for the html2rss command-line interface

import { Code } from "@astrojs/starlight/components";

This page documents the html2rss command-line interface (CLI).

For detailed documentation on the Ruby API, please refer to the official YARD documentation.

📚 View the Ruby API Docs on rubydoc.info

Commands

The html2rss executable is the primary way to interact with the gem from your terminal.

Scrape

Automatically discovers items from a page and prints the generated RSS or JSON Feed to stdout. (html2rss auto is a Thor alias.)

<Code code={html2rss scrape https://example.com/articles ; \\ html2rss scrape https://example.com/articles --format jsonfeed ; \\ html2rss scrape https://example.com/articles --limit 10 ; \\ BOTASAURUS_SCRAPER_URL="http://localhost:4010" html2rss scrape https://example.com/protected --strategy botasaurus ; \\ html2rss scrape https://example.com/articles --items_selector ".post-card" ; \\ html2rss scrape https://example.com/articles --explain ; \\ html2rss scrape --input sample.html ; \\ html2rss scrape https://example.com/articles --input sample.html} lang="bash" />

Command: html2rss scrape [URL]

Available options:

  • --strategy: Optional request strategy (auto, default, botasaurus, local_file). Defaults to auto, which chooses an available fetch path based on scrape results. Use a concrete value only for diagnosis or strategy-specific controls.
  • --format: Output format for the auto-sourced feed (rss or jsonfeed). Defaults to rss.
  • --limit: Maximum number of articles to extract during discovery (defaults to 25).
  • --items_selector: Optional CSS selector hint for item extraction.
  • --max-redirects: Maximum redirects to follow per request.
  • --max-requests: Maximum requests to allow for this feed build (defaults to 4 for auto discovery).
  • --input: Local HTML file path to read input from without making network requests. When provided without a URL, html2rss attempts to detect the base URL from <link rel="canonical"> or OpenGraph metadata in the HTML.
  • --explain: Print Html2rss::Status JSON to stderr (version, dedup_dropped, and when present scraper_tallies, selected_strategy, attempt_count, strategy_attempts, admission_drops, entry_url, scrape_url, entry_resolution). Stdout stays the RSS or JSON Feed.

URL surface guidance for scrape

scrape works best when the input URL already exposes a server-rendered list of entries.

  • High-success surfaces:
    • newsroom or press listing pages
    • blog/category/tag listing pages
    • changelog/release notes/update listing pages
    • paginated archive/list views
  • Low-success surfaces:
    • generic homepages with heavy promo/navigation chrome
    • search results pages
    • client-rendered app shells (#app, #root, #__next, etc.)

When possible, pass a direct listing/update URL instead of a top-level homepage or app entrypoint.

Failure outcomes

When no extractable items are found, scrape classifies likely causes instead of only returning a generic message:

  • blocked surface likely (anti-bot or interstitial):
    • try a more specific public listing URL or configure BOTASAURUS_SCRAPER_URL
  • app-shell surface detected:
    • switch to a direct listing/update URL or configure BOTASAURUS_SCRAPER_URL
  • unsupported extraction surface for auto mode:
    • switch to listing/changelog/category URLs
    • use explicit selectors in a feed config

Known anti-bot interstitial responses (for example Cloudflare challenge pages) are surfaced explicitly as blocked-surface errors.

If auto exhausts the available fetch paths without items, html2rss raises:

  • No RSS feed items extracted after auto fallback ...

If failures continue after URL/surface fixes, ensure BOTASAURUS_SCRAPER_URL is set so auto can use the companion scraper.

Start by changing the input URL to a direct listing/update page, then move to explicit selectors if needed.

Botasaurus Environment Requirement (CLI)

botasaurus is an explicit strategy option for CLI usage and requires BOTASAURUS_SCRAPER_URL:

<Code code={BOTASAURUS_SCRAPER_URL="http://localhost:4010" \\ html2rss scrape https://example.com/updates --strategy botasaurus} lang="bash" />

If you see a Botasaurus configuration error, check:

  • BOTASAURUS_SCRAPER_URL is set
  • BOTASAURUS_SCRAPER_URL is a valid URL
  • the Botasaurus scrape API is reachable from the shell environment running html2rss

Apply

Loads a YAML config, builds the feed, and prints the RSS XML to stdout. (html2rss feed is a Thor alias.)

<Code code={html2rss apply single.yml ; \\ html2rss apply feeds.yml my-first-feed ; \\ html2rss apply single.yml --strategy auto ; \\ BOTASAURUS_SCRAPER_URL="http://localhost:4010" html2rss apply single.yml --strategy botasaurus ; \\ html2rss apply single.yml --max-redirects 5 --max-requests 6 ; \\ html2rss apply single.yml --params id:42 foo:bar ; \\ html2rss apply single.yml --input sample.html} lang="bash" />

Command: html2rss apply YAML_FILE [feed_name]

Available options:

  • --strategy: Request strategy override (auto, default, botasaurus, local_file).
  • --params: Dynamic parameters passed as key-value pairs (e.g. --params id:42 section:news).
  • --max-redirects: Maximum redirects to follow per request.
  • --max-requests: Total request budget allowed for this feed build.
  • --input: Local HTML file path to read input from instead of making a network request.

The CLI keeps strategy as a top-level override and writes runtime request limits into the generated config under request.

Capture

Analyzes a URL using auto-source structural analysis and prints a reusable YAML feed configuration (items selector + enhance: true) to stdout.

<Code code={html2rss capture https://example.com/articles ; \\ html2rss capture https://example.com/articles --strategy botasaurus ; \\ html2rss capture https://example.com/articles --items_selector ".post-card" ; \\ html2rss capture https://example.com/articles --explain ; \\ html2rss capture --input sample.html} lang="bash" />

Command: html2rss capture [URL]

Available options:

  • --strategy: Optional request strategy (auto, default, botasaurus, local_file). Defaults to auto.
  • --items_selector: Optional CSS selector hint for item extraction.
  • --limit: Maximum number of articles to keep (defaults to 25).
  • --max-redirects: Maximum redirects to follow per request.
  • --max-requests: Maximum requests to allow for this feed build.
  • --input: Local HTML file path to read input from without making network requests.
  • --explain: Print capture quality JSON to stderr (articles_count, channel_title, has_selectors, segment_strategy, selected_strategy, admission_drops). Stdout stays YAML.

Capture records the concrete strategy that produced the draft as a top-level strategy: so later html2rss apply can reproduce that fetch path.

MCP

Starts the Model Context Protocol (MCP) server for integration with AI clients and agent workflows (Cursor, Claude Desktop, Copilot).

<Code code={html2rss mcp ; \\ html2rss mcp --transport stdio ; \\ html2rss mcp --transport http --port 8080} lang="bash" />

Command: html2rss mcp

Available options:

  • --transport: MCP transport protocol (stdio or http). Defaults to stdio.
  • --port: Port for HTTP transport (default: 8080). Bound strictly to 127.0.0.1 (loopback only).

HTTP transport requires the rack, rackup, and webrick gems. MCP daemon logging (stderr, LOG_LEVEL) is documented in the MCP Server Reference. See also the AI Agent Workflows Guide for client setup and tool schemas.

Schema

Prints or writes the exported JSON Schema for the current gem version.

<Code code={html2rss schema ; \\ html2rss schema --no-pretty ; \\ html2rss schema --write tmp/html2rss-config.schema.json} lang="bash" />

Command: html2rss schema

Available options:

  • --pretty / --no-pretty: Pretty-print the schema JSON (default: true).
  • --write PATH: Write the schema JSON directly to the given file path, creating any missing parent directories.

Validate

Validates a config with the runtime validator without generating a feed.

<Code code={html2rss validate single.yml ; \\ html2rss validate feeds.yml my-first-feed ; \\ html2rss validate single.yml --params id:42} lang="bash" />

Command: html2rss validate YAML_FILE [feed_name]

Available options:

  • --params: Dynamic parameters to interpolate before validation (default: {}).

Help

Displays the help message with available commands and options.

Command: html2rss help

Version

Displays the installed version of html2rss.

Command: html2rss --version