webcat
The repo for the Web Cat MCP Server - A simple and reliable search server
Documentation
WebCat MCP Server
Web search and content extraction for AI models via Model Context Protocol (MCP)
Quick Start
Docker (Recommended)
# Run with Docker (no setup required)
docker run -p 8000:8000 tmfrisinger/webcat:latest
# With Serper API key for premium search
docker run -p 8000:8000 -e SERPER_API_KEY=your_key tmfrisinger/webcat:latest
# With authentication enabled
docker run -p 8000:8000 -e WEBCAT_API_KEY=your_token tmfrisinger/webcat:latestSupports: linux/amd64, linux/arm64 (Intel/AMD, Apple Silicon, AWS Graviton)
Local Development
cd docker
python -m pip install -e ".[dev]"
# Start MCP server with auto-reload
make dev
# Or run directly
python mcp_server.pyWhat is WebCat?
WebCat is an MCP (Model Context Protocol) server that provides AI models with:
- ๐ Web Search - Serper API (premium) or DuckDuckGo (free fallback)
- ๐ Content Extraction - Serper scrape API (premium) or Trafilatura (free fallback)
- ๐ Modern HTTP Transport - Streamable HTTP with JSON-RPC 2.0
- ๐ณ Multi-Platform Docker - Works on Intel, ARM, and Apple Silicon
- ๐ฏ Composite Tool - Single SERPER_API_KEY enables both search + scraping
Built with FastMCP, Serper.dev, and Trafilatura for seamless AI integration.
Features
- โ Optional Authentication - Bearer token auth when needed, or run without (v2.3.1)
- โ Composite Search Tool - Single Serper API key enables both search + scraping
- โ Automatic Fallback - Search: Serper โ DuckDuckGo | Scraping: Serper โ Trafilatura
- โ Premium Scraping - Serper's optimized infrastructure for fast, clean content extraction
- โ Smart Content Extraction - Returns markdown with preserved document structure
- โ MCP Compliant - Works with Claude Desktop, LiteLLM, and other MCP clients
- โ Parallel Processing - Fast concurrent scraping
- โ Multi-Platform Docker - Linux (amd64/arm64) support
Installation & Usage
Docker Deployment
# Quick start - no configuration needed
docker run -p 8000:8000 tmfrisinger/webcat:latest
# With environment variables
docker run -p 8000:8000 \
-e SERPER_API_KEY=your_key \
-e WEBCAT_API_KEY=your_token \
tmfrisinger/webcat:latest
# Using docker-compose
cd docker
docker-compose upLocal Development
cd docker
python -m pip install -e ".[dev]"
# Configure environment (optional)
echo "SERPER_API_KEY=your_key" > .env
# Development mode with auto-reload
make dev # Start MCP server with auto-reload
# Production mode
make mcp # Start MCP serverAvailable Endpoints
| Endpoint | Description |
|---|---|
| `http://localhost:8000/health` | ๐ Health check |
| `http://localhost:8000/status` | ๐ Server status |
| `http://localhost:8000/mcp` | ๐ ๏ธ MCP protocol endpoint (Streamable HTTP with JSON-RPC 2.0) |
Configuration
Environment Variables
| Variable | Default | Description |
|---|---|---|
| `SERPER_API_KEY` | *(none)* | Serper API key for premium search (optional, falls back to DuckDuckGo if not set) |
| `PERPLEXITY_API_KEY` | *(none)* | Perplexity API key for deep research tool (optional, get at https://www.perplexity.ai/settings/api) |
| `WEBCAT_API_KEY` | *(none)* | Bearer token for authentication (optional, if set all requests must include `Authorization: Bearer `) |
| `PORT` | `8000` | Server port |
| `LOG_LEVEL` | `INFO` | Logging level (DEBUG, INFO, WARNING, ERROR) |
| `LOG_DIR` | `/tmp` | Log file directory |
| `MAX_CONTENT_LENGTH` | `1000000` | Maximum characters to return per scraped article |
Get API Keys
Serper API (for web search + scraping):
1. Visit serper.dev
2. Sign up for free tier (2,500 searches/month + scraping)
3. Copy your API key
4. Add to `.env` file: `SERPER_API_KEY=your_key`
5. Note: One API key enables both search AND content scraping!
Perplexity API (for deep research):
1. Visit perplexity.ai/settings/api
2. Sign up and get your API key
3. Copy your API key
4. Add to `.env` file: `PERPLEXITY_API_KEY=your_key`
Enable Authentication (Optional)
To require bearer token authentication for all MCP tool calls:
1. Generate a secure random token: `openssl rand -hex 32`
2. Add to `.env` file: `WEBCAT_API_KEY=your_token`
3. Include in all requests: `Authorization: Bearer your_token`
Note: If `WEBCAT_API_KEY` is not set, no authentication is required.
MCP Tools
WebCat exposes these tools via MCP:
| Tool | Description | Parameters |
|---|---|---|
| `search` | Search web and extract content | `query: str`, `max_results: int` |
| `scrape_url` | Scrape specific URL | `url: str` |
| `health_check` | Check server health | *(none)* |
| `get_server_info` | Get server capabilities | *(none)* |
Architecture
MCP Client (Claude, LiteLLM)
โ
FastMCP Server (Streamable HTTP with JSON-RPC 2.0)
โ
Authentication (optional bearer token)
โ
Search Decision
โโ Serper API (premium) โ Serper Scrape API (premium)
โโ DuckDuckGo (free) โ Trafilatura (free)
โ
Markdown ResponseTech Stack:
- FastMCP - MCP protocol implementation with modern HTTP transport
- JSON-RPC 2.0 - Standard protocol for client-server communication
- Serper API - Google-powered search + optimized web scraping
- Trafilatura - Fallback content extraction (removes navigation/ads)
- DuckDuckGo - Free search fallback
Testing
cd docker
# Run all unit tests
make test
# OR
python -m pytest tests/unit -v
# With coverage report
make test-coverage
# OR
python -m pytest tests/unit --cov=. --cov-report=term --cov-report=html
# CI-safe tests (no external dependencies)
python -m pytest -v -m "not integration"
# Run specific test file
python -m pytest tests/unit/services/test_content_scraper.py -vCurrent test coverage: 70%+ across all modules (enforced in CI)
Development
# First-time setup
make setup-dev # Install all dependencies + pre-commit hooks
# Development workflow
make dev # Start server with auto-reload
make format # Auto-format code (Black + isort)
make lint # Check code quality (flake8)
make test # Run unit tests
# Before committing
make ci-fast # Quick validation (~30 seconds)
# OR
make ci # Full validation with security checks (~2-3 minutes)
# Code quality tools
make format-check # Check formatting without changes
make security # Run bandit security scanner
make audit # Check dependency vulnerabilitiesPre-commit Hooks:
Hooks run automatically on `git commit` to ensure code quality. Install with `make setup-dev`.
Project Structure
docker/
โโโ mcp_server.py # Main MCP server (FastMCP)
โโโ cli.py # CLI interface for server modes
โโโ health.py # Health check endpoint
โโโ api_tools.py # API tooling utilities
โโโ clients/ # External API clients
โ โโโ serper_client.py # Serper API (search + scrape)
โ โโโ duckduckgo_client.py # DuckDuckGo fallback
โโโ services/ # Core business logic
โ โโโ search_service.py # Search orchestration
โ โโโ content_scraper.py # Serper scrape โ Trafilatura fallback
โโโ tools/ # MCP tool implementations
โ โโโ search_tool.py # Search tool with auth
โโโ models/ # Pydantic data models
โ โโโ domain/ # Domain entities (SearchResult, etc.)
โ โโโ responses/ # API response models
โโโ utils/ # Shared utilities
โ โโโ auth.py # Bearer token authentication
โโโ endpoints/ # FastAPI endpoints
โโโ tests/ # Comprehensive test suite
โ โโโ unit/ # Unit tests (mocked dependencies)
โ โโโ integration/ # Integration tests (external deps)
โโโ pyproject.toml # Project config + dependenciesSearch Quality Comparison
| Feature | Serper API | DuckDuckGo |
|---|---|---|
| Cost | Paid (free tier available) | Free |
| Quality | โญโญโญโญโญ Excellent | โญโญโญโญ Good |
| Coverage | Comprehensive (Google-powered) | Standard |
| Speed | Fast | Fast |
| Rate Limits | 2,500/month (free tier) | None |
Docker Multi-Platform Support
WebCat supports multiple architectures for broad deployment compatibility:
# Build locally for multiple platforms
cd docker
./build.sh # Builds for linux/amd64 and linux/arm64
# Manual multi-platform build and push
docker buildx build --platform linux/amd64,linux/arm64 \
-t tmfrisinger/webcat:2.3.2 \
-t tmfrisinger/webcat:latest \
-f Dockerfile --push .
# Verify multi-platform support
docker buildx imagetools inspect tmfrisinger/webcat:latestAutomated Releases:
Push a version tag to trigger automated multi-platform builds via GitHub Actions:
git tag v2.3.2
git push origin v2.3.2Limitations
- Text-focused: Optimized for article content, not multimedia
- No JavaScript: Cannot scrape dynamic JS-rendered content (uses static HTML)
- PDF support: Detection only, not full extraction
- Python 3.11 required: Not compatible with 3.10 or 3.12
- External API limits: Subject to Serper API rate limits (2,500/month free tier)
Contributing
Contributions welcome! Please:
1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Ensure `make ci` passes
5. Submit a Pull Request
See CLAUDE.md for development guidelines and architecture standards.
License
MIT License - see LICENSE file for details.
Links
- GitHub: github.com/Kode-Rex/webcat
- MCP Spec: modelcontextprotocol.io
- Serper API: serper.dev
Version 2.3.2 | Built with FastMCP, FastAPI, Readability, and html2text
Frequently asked questions
What is webcat?
webcat is The repo for the Web Cat MCP Server - A simple and reliable search server
How do I install webcat?
Open the GitHub repository and follow its README. Most MCP servers are added to your client's MCP config, then called by your agent.
Is webcat open source?
Yes โ it is hosted on GitHub at https://github.com/Kode-Rex/webcat and has 1 stars.
Related MCP tools
An LLM agent that conducts deep research (local and web) on any given topic and generates a long report with citations. Built for the Model Context Protocol to
An MCP server that autonomously evaluates web applications. Python-based implementation. Trusted by 1100+ developers. Trusted by 1100+ developers.
Model Context Protocol with Neo4j Python-based implementation. Trusted by 700+ developers. Trusted by 700+ developers. Trusted by 700+ developers.
MCP Aggregator, Orchestrator, Middleware, Gateway in one docker TypeScript-based implementation. Trusted by 1400+ developers.
A powerful coding agent toolkit providing semantic retrieval and editing capabilities (MCP server & other integrations) Python-based implementation.
Expose your FastAPI endpoints as Model Context Protocol (MCP) tools, with Auth! Python-based implementation. Trusted by 11000+ developers.
Run your own MCP server? See who uses it and what to fix.
Measure it with TrackMCP