rodricios/wxpath
wxpath - declarative web crawling with XPath; a Web Query Language (WQL)
Combines XPath 3.1 with custom `url()` and `///` operators to express multi-level web graph traversal in a single declarative expression, eliminating imperative crawl loops. Executes breadth-first concurrent requests with built-in deduplication, streaming results as discovered, and supports structured extraction via XPath 3.1 features like `map{}` for JSON-ready output. Includes an interactive TUI (powered by Textual), respects `robots.txt` by default, and integrates with LangChain for RAG pipelines.
108 stars and 177 monthly downloads. Available on PyPI.
Stars
108
Forks
5
Language
Python
License
MIT
Category
Last pushed
Feb 28, 2026
Monthly downloads
177
Commits (30d)
0
Dependencies
4
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/rodricios/wxpath"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts...
kreuzberg-dev/html-to-markdown
High performance and CommonMark compliant HTML to Markdown converter. Maintained by the...
lightfeed/extractor
Using LLMs and AI browser automation to robustly extract web data
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file...