henu-wang/ai-crawlers-reference
Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples
Stars
1
Forks
—
Language
HTML
License
—
Category
Last pushed
Mar 09, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/henu-wang/ai-crawlers-reference"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
ai-search-guru/getcito-worlds-first-open-source-aio-aeo-or-geo-tool
GetCito the World's First & Only Open Source AI Search Optimization (AIO) / Answer Engine...
mverab/eGEOagents
Generative Engine Optimization skills for AI agents. Optimize content for ChatGPT, Perplexity,...
Aryanpanwar10005/seo-geo-optimizer
Universal SEO & GEO skill for AI IDEs — Google, Bing, AI Overviews, Bing Copilot, ChatGPT,...
multivmlabs/aeo.js
Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude,...
AEO-Content-Inc/aeorank
Score any website for AI engine visibility across 26 criteria. Zero API keys, under 10 seconds.