ai-crawlers

Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude, Perplexity & AI search engines. Generates llms.txt, robots.txt, sitemap, JSON-LD & more.

Automate web workflows on real browser tabs without hosting a browser farm.

Curated GEO platforms, repositories, and verification scripts for AI-era discoverability.

Automatically generate llms.txt and llms-full.txt files for Laravel applications to help AI models and LLM crawlers understand your website.

Scope, audit, harden, and retest web projects with AI coding agents and reproducible evidence.

Check whether a website is reachable and readable by AI crawlers, and score it. Two scores, never averaged. Blocking model training costs no points.

Server-side AI crawler & bot traffic tracking for Laravel — reports ChatGPT, Claude, Googlebot, GPTBot & co. to DataFast. Unofficial PHP port of @datafast/ai-crawl.

🤖 Generate optimized robots.txt files for AI search engine crawlers (GPTBot, PerplexityBot, ClaudeBot, and more)

See your site the way search engines and AI see it.

Curated list of AI visibility / GEO / AEO resources: tools, AI crawler datasets, agent skills, llms.txt, MCP servers, research papers, and guides.

Claude Code skill that generates and audits ai.txt, robots.txt (AI-crawler rules), llms.txt, and TDM opt-out files for websites.

Make your website work for agents - audit and fix how AI crawlers, answer engines, and agents discover, read, cite, and act on a site. Live probe + codebase fixes, scored across 6 pillars.

AI-Crawler-Blocklist publishes AI crawler blocklists and deployment-ready firewall snippets from official operator-published sources. It separates verified IP ranges, user-agent rules, robots.txt controls, and watch lists so site operators can choose the right enforcement level without mixing signal quality.

Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples

Smart prerendering for WordPress — make your content visible to search engines, social bots, and AI crawlers. Zero config. 29+ bots. ~200ms.

Define AI permissions once. Emit the rules today's AI web actually uses. One manifest → robots.txt, AIPREF headers, X-Robots-Tag, Google-Extended, TDMRep.

LogWerk is a lightweight, offline-first server log analyzer dashboard that runs entirely in your web browser.

Free, deterministic GEO audit that scores whether AI answer engines (ChatGPT, Perplexity, Claude, Google AI Overviews) can retrieve, read, and quote your website

Evidence-led SEO and AI-search skill for diagnosing, implementing, releasing, and verifying real sites.

MOVED: This tool is now part of the aah.monster ecosystem.