Sandbox
9 repos for crawler · Claude Code · DocsClear
Sriram-PR/
doc-scraper

Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data).

99
adityaarsharma/
librecrawl-technical-seo-audit-mcp

The AI-native technical SEO crawler. Open-source MCP server for Claude / Cursor / Codex — 37 tools, 50+ checks, unlimited pages, WAF detection, ephemeral by design. Built on LibreCrawl. MIT.

39
tigerless-labs/
seo-ops

SEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.

136
pinkpixel-dev/
web-scout-mcp

An MCP server providing web search and content extraction capabilities. Integrates DuckDuckGo search functionality and URL content extraction into your MCP environment, enabling AI assistants to search the web and extract webpage content.

133
just-every/
mcp-read-website-fast

Quickly reads webpages and converts to markdown for fast, token efficient web scraping

161

🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.

7.4k

The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

5.2k

Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.

98