Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.

Sitewalk: SEO Audit & Site Crawler. 30+ technical SEO checks. Use it in ChatGPT with no install, or self-host the open-source crawler and MCP server for Claude Code, Cursor and Codex. SEO growth agent included. Powered by LibreCrawl. MIT.

Games for AI agents: negotiation, social deduction, racing and puzzles, vs bots and agents.

Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

Agent skills for squirrelscan, the website QA tool for your coding agent (300+ audit rules).

Lightweight scraper for Google News

CrawlForge MCP Server - Professional Model Context Protocol server with 31 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama

🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.

MCP server that gives AI agents the web: fetch, search, crawl and screenshot from one local Rust binary. Zero API keys, zero accounts, Chrome-true TLS.

The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

Local MCP for web search, direct-URL fetch, site crawling, docs export, and bounded smartbrowser local browser actions.

Model Context Protocol (MCP) Server for Graphlit Platform

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.

Deterministic Zero-LLM Web Compiler, Scraper, Crawler, and Site Mapper for AI Agents.

Checks a site for AI crawlers: which of 11 AI crawlers its robots.txt lets in, whether its llms.txt follows llmstxt.org, and what its home page says without scripts. Findings with evidence; no scores, no model calls, no telemetry.

Free, rules-based checks for e-commerce product pages: AI shopping readiness (structured data, reviews, return and shipping terms, GTINs, AI crawler access), Google Merchant Center compliance, store SEO audit and Shopify checkout deadlines. No AI calls, nothing stored.

Enables AI assistants to diagnose and fix technical SEO, content and generative-engine-optimization issues in one conversation by bundling 102 tools spanning Google Search Console, Analytics 4, PageSpeed/CrUX, structured data, llms.txt, AI-crawler access, WordPress and GitHub.

Enables AI agents to run web searches and fetch page content by driving a real Chromium browser over CDP, so every request carries an existing profile's cookies, login state, and fingerprint. It also lets agents click elements, fill and submit forms, read the current tab, and check browser/CDP status, returning accurate results from logged-in pages without being treated as an anonymous crawler.

Automated index of Claude Code plugins and adoption metrics across GitHub repositories, powered by a Node.js/TypeScript crawler

MCP server that checks a website's AI-search (GEO) readiness: robots.txt access for 16 AI crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended, Bingbot...), JSON-LD schema, llms.txt, sitemap and basic SEO tags. Python, stdio, MIT.

Live AEO, GEO, JSON-LD Schema, robots.txt AI crawler, and Agent Readiness diagnostic auditor.

Free AI crawler, schema, llms.txt and answer-readiness checks for public web pages. No API key.