spider-rs/spider
Web crawler and scraper for Rust
Supports multiple rendering engines (HTTP, Chrome DevTools Protocol, WebDriver) with feature-gated compilation for minimal binary size, plus built-in proxy rotation, caching layers (memory/disk/hybrid), and anti-bot fingerprinting. Leverages Tokio's async runtime and lock-free data structures for concurrent crawling of 100k+ pages, with optional io_uring on Linux. Integrates with OpenAI/Gemini for content analysis, Spider Cloud for managed proxy/CAPTCHA bypass, and includes a multimodal LLM agent for web automation tasks.
2,323 stars. Actively maintained with 171 commits in the last 30 days.
Stars
2,323
Forks
183
Language
Rust
License
MIT
Category
Last pushed
Mar 13, 2026
Commits (30d)
171
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/agents/spider-rs/spider"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Compare
Related agents
Dicklesworthstone/pi_agent_rust
High-performance AI coding agent CLI written in Rust with zero unsafe code
RightNow-AI/openfang
Open-source Agent Operating System
liquidos-ai/AutoAgents
A multi-agent framework written in Rust that enables you to build, deploy, and coordinate...
Ataraxy-Labs/sem
Semantic version control CLI. Entity-level diff, blame, graph, and impact analysis for code. 16...
sayiir/sayiir
Sayiir — simple, embeddable durable workflow engine in Rust, node.js/python bindings....