oxylabs/ai-crawler-py
Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.
Built on the Oxylabs AI Studio platform, it uses LLM-guided crawling with intelligent URL prioritization to explore sites dynamically—eliminating the need for static CSS/XPath selectors. Supports both Markdown and JSON output with optional automatic schema generation from natural language, alongside JavaScript rendering and geo-targeting capabilities via Python SDK.
2,764 stars.
Stars
2,764
Forks
12
Language
—
License
—
Category
Last pushed
Oct 13, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/agents/oxylabs/ai-crawler-py"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
vakra-dev/reader
Open-source, production-grade web scraping engine built for LLMs. Scrape and crawl the entire...
joaobenedetmachado/scrapit
A (really) easy way to web scrape
firecrawl/open-scouts
🔥 AI-powered web monitoring platform. Create automated scouts that search the web and send email...
memvid/maw
Crawl any website into a single searchable file. Query it forever, offline.
BrowserCash/teracrawl
High-performance web crawler API optimized for LLMs. Turn any search or website into clean...