Skip to content

troubleshooting.mdx

Latest commit

 

History

History
183 lines (122 loc) · 9.04 KB

troubleshooting.mdx

File metadata and controls

183 lines (122 loc) · 9.04 KB
Raw
title Troubleshooting
description This guide provides solutions to common issues encountered when using html2rss.

import { Code } from "@astrojs/starlight/components";

This guide provides solutions to common issues encountered when using html2rss.

Essential Tools

Your browser's developer tools are essential for troubleshooting. Use them to inspect the HTML structure of a webpage and find the correct CSS selectors.

  • To open: Right-click an element on a webpage and select "Inspect" or "Inspect Element."

Common Issues (Ruby Gem / CLI)

auto Picks The Wrong Surface Or Finds No Items

The auto flow is URL-surface sensitive.

  • Higher success inputs:
    • newsroom/press listing URLs
    • category/tag/listing/archive URLs
    • changelog/release/update listing URLs
  • Lower success inputs:
    • generic homepages
    • search result pages
    • client-rendered app-shell entrypoints

If extraction quality is poor, switch to a more specific listing/update URL before tuning selectors. html2rss scrape URL --explain prints admission drop counts on stderr so you can see why items were dropped (junk titles, non-article destinations) without mixing JSON into the feed.

Empty Feeds

If your feed is empty, check the following:

  • URL: Ensure the url in your configuration is correct and accessible.
  • items.selector: Verify that the items.selector matches the elements on the page.
  • Website Changes: Websites change their HTML structure frequently. Your selectors may be outdated.
  • JavaScript Content: If the content is loaded via JavaScript, use a browser-based rendering strategy.
  • Authentication: Some sites require authentication — check if you need to add headers or use a different strategy.

No scrapers found Failure Taxonomy (auto)

auto classifies no-scraper failures with actionable hints:

  • Blocked surface likely (anti-bot or interstitial):
    • try a more specific public listing URL
  • App-shell surface detected:
    • target a direct listing/update page instead of homepage/shell entrypoint
  • Unsupported extraction surface for auto mode:
    • switch to listing/changelog/category URLs
    • or use explicit selectors in YAML config

Known anti-bot interstitial patterns (for example Cloudflare or DataDome challenge pages) are surfaced as blocked-surface errors instead of silent empty extraction results.

When auto exhausts the available fetch paths without items, html2rss raises No RSS feed items extracted after auto fallback ....

If failures continue after URL/surface fixes, configure BOTASAURUS_SCRAPER_URL so auto can use the companion scraper. Pin --strategy botasaurus only for browser-specific diagnosis or controls.

DataDome And Similar Challenges

DataDome (and similar) challenge pages are classified as a blocked surface. Prefer:

  1. A direct public listing/update URL instead of a challenge-prone entrypoint
  2. strategy: botasaurus with BOTASAURUS_SCRAPER_URL so the browser tier can complete navigation
  3. A session that already has resolved challenge cookies when the site still blocks automated browsers

Do not expect static HTTP fetches to succeed against active DataDome interstitials.

Botasaurus Setup & Diagnostics

If you receive a Botasaurus configuration error or connection failure:

  1. Confirm the Botasaurus scrape API service is running and reachable from the machine running html2rss.
  2. Confirm BOTASAURUS_SCRAPER_URL points at your Botasaurus instance (for example http://localhost:4010).

Example local startup via Docker:

<Code code={docker run -d --rm -p 4010:4010 html2rss/botasaurus-scrape-api:latest} lang="bash" />

Then run with:

<Code code={BOTASAURUS_SCRAPER_URL="http://localhost:4010" \\ html2rss scrape https://example.com/updates --strategy botasaurus} lang="bash" />

Configuration Errors

Common configuration-related errors:

  • UnsupportedResponseContentType: The website returned content that html2rss cannot parse (not HTML or JSON). Supported compressed HTML and bodies that clearly identify themselves as HTML are accepted even when response headers are incomplete.
  • UnknownStrategy: The specified strategy is not available. Use auto, default, botasaurus, or local_file. Prefer auto unless you need a fixed mode.
  • BOTASAURUS_SCRAPER_URL is required for strategy=botasaurus.: Set BOTASAURUS_SCRAPER_URL to your Botasaurus scrape API base URL when using --strategy botasaurus.
  • BOTASAURUS_SCRAPER_URL is invalid: Fix the URL format and retry.
  • Configuration must include at least 'selectors' or 'auto_source': You need to specify either manual selectors or enable auto-source.
  • stylesheet.type invalid: Only text/css and text/xsl are supported for stylesheets.
  • request.botasaurus validation: Unknown keys are rejected. Valid scroll, window size, and wait-timeout shapes are documented in Strategy — Botasaurus.

Missing Item Parts

If parts of your items (e.g., title, link) are missing, check the following:

  • Selector: Ensure the selector for the missing part is correct and relative to the items.selector.
  • Extractor: Verify that you are using the correct extractor (e.g., text, href, attribute).
  • Dynamic Content: Configure BOTASAURUS_SCRAPER_URL and keep auto so html2rss can render the page when needed. Pin --strategy botasaurus only for browser-specific diagnosis or controls.

Date/Time Parsing Errors

If you are having issues with date/time parsing, check the following:

  • Date Format: The parse_time post-processor automatically detects common date formats using Ruby's Time.parse. Ensure your date strings are in a recognizable format.
  • time_zone: Specify the correct time_zone if the website uses a specific time zone.

html2rss Command Not Found

If you are getting a "command not found" error, try the following:

  • Re-install: Re-install html2rss to ensure it is installed correctly: gem install html2rss.
  • Check PATH: Ensure that the directory where Ruby gems are installed is in your system's PATH.

Web Application Issues (html2rss-web)

Instance Won’t Start

  • Verify Docker is installed and running: <Code code={docker --version} lang="bash" />
  • Check logs for errors: <Code code={docker compose logs} lang="bash" />
  • Ensure the app port (default compose binding: 4000) isn’t already in use: <Code code={lsof -i :4000} lang="bash" />
  • If the app exits immediately in production, check that HTML2RSS_SECRET_KEY is set.

Can’t Access the Web Interface

  • Confirm your firewall allows traffic on port 4000 or your reverse-proxy ports
  • Try accessing via the server’s IP instead of a domain name
  • Double-check that containers are running: <Code code={docker compose ps} lang="bash" />

Authentication Errors

  • 401 Unauthorized when creating feeds: Re-enter the same access token you set as HTML2RSS_ACCESS_TOKEN for your instance.
  • 403 Forbidden when creating feeds: Automatic feed generation may be disabled (AUTO_SOURCE_ENABLED=false) or the requested URL may not be allowed for the authenticated account.
  • 500 Internal Server Error: Check the application logs for detailed error information.
  • Health endpoint failures: Start with GET /api/v1/health/ready for standard readiness checks. Use GET /api/v1/health/live for process-alive checks, and use authenticated GET /api/v1/health only when you intentionally configured HEALTH_CHECK_TOKEN.

Rate Limiting & Timeouts

  • 429 Too Many Requests: The client IP has exceeded the rate limit. Check the RATE_LIMIT_MAX_REQUESTS and RATE_LIMIT_WINDOW_SECONDS settings. The response includes a Retry-After header indicating how many seconds to wait before retrying.
  • 503 Service Unavailable: The server is busy or the request timed out at the server layer (e.g. REQUEST_TIMEOUT_SECONDS). Returns a Retry-After header matching RETRY_AFTER_TIMEOUT_SECONDS.
  • 504 Gateway Timeout: The target website or scraper took too long to respond (exceeding HTML2RSS_TOTAL_TIMEOUT_SECONDS).

Feed Problems

  • Try a more specific listing, newsroom, changelog, or updates URL before changing infrastructure
  • If your first-run stack includes Botasaurus, ensure the botasaurus service is running
  • Check your feed YAML (and registry.id if you used registry add) for typos or invalid selectors
  • Look for parsing errors in the logs: <Code code={docker compose logs html2rss-web} lang="bash" />

Tips & Tricks

  • Mobile Redirects: Check that the channel URL does not redirect to a mobile page with a different markup structure.
  • curl and pup: For static sites, use curl and pup to quickly find selectors: curl URL | pup.
  • CSS Selectors: For a comprehensive overview of CSS selectors, see the W3C documentation.

Still Stuck?