raintree-technology/docpull
Crawl any website and convert it to clean, AI-ready Markdown — async Python CLI with MCP support, crawl profiles, caching, and RAG-optimized output
Uses pattern-based URL filtering (include/exclude paths) and preset profiles (RAG, mirror, quick) to optimize for different use cases, with streaming deduplication for memory efficiency. Async Python architecture with optional Playwright integration for JavaScript-heavy sites, YAML frontmatter output, and built-in security controls (robots.txt enforcement, private IP blocking). Exposes both CLI and async Python API with event streaming for progress tracking and incremental caching with configurable TTL.
Available on PyPI.
Stars
20
Forks
1
Language
Python
License
MIT
Category
Last pushed
Feb 06, 2026
Monthly downloads
69
Commits (30d)
0
Dependencies
10
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/raintree-technology/docpull"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts...
kreuzberg-dev/html-to-markdown
High performance and CommonMark compliant HTML to Markdown converter. Maintained by the...
lightfeed/extractor
Using LLMs and AI browser automation to robustly extract web data
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file...