MCP Web Scraper

MCP Web Scraper is an MCP server that connects Model Context Protocol clients to a Playwright-powered web automation and scraping environment. Designed for developers, researchers, and autonomous AI agents, it enables automated browser interactions, structured content extraction, and dynamic page navigation. The server integrates 29 distinct browser tools providing parity with Microsoft Playwright MCP, covering click events, text input, tab management, drag-and-drop actions, network monitoring, and PDF export. Additionally, it features automated cookie consent handling across multiple frameworks and languages, allowing agents to access web content without manual modal dismissal. Content extraction is backed by machine learning pattern recognition and a persistent SQLite cache to retain extraction results and accelerate repeated queries across web domains and publishing platforms like Medium, Substack, and Dev.to. The server operates as a standalone service via Docker or Node.js, exposing standard MCP protocol endpoints over Server-Sent Events or local execution. Users can capture full-page screenshots, inspect console logs, execute custom JavaScript, and extract articles directly into text, HTML, or Markdown formats for downstream language model processing.

Category: Browser & Web Automation

Tags: automation, caching, playwright, web scraping

Visit MCP Web Scraper

How to install and configure MCP Web Scraper

1. Pull and run the server container using Docker: bash docker run -p 3001:3001 descoped/mcp-web-scraper Alternatively, clone and run it with Node.js: bash git clone https://github.com/descoped/mcp-web-scraper.git cd mcp-web-scraper npm install npm run build npm start 2. Confirm the server is reachable by running a health check: bash curl http://localhost:3001/health 3. Connect your MCP client (such as Claude Desktop or custom TypeScript clients) to the running server instance at http://localhost:3001 as outlined in the repository README.

What you can do with MCP Web Scraper

• Extract clean article text, metadata, and author details in Markdown or HTML from publishing platforms like Substack and Medium. - Bypass cookie consent banners automatically across international websites before capturing full-page screenshots or extracting content. - Execute end-to-end browser interactions such as clicking, typing, selecting dropdowns, and managing multiple browser tabs. - Monitor network traffic and track HTTP requests, response headers, and status codes during automated web sessions. - Save web pages as PDF documents or retrieve accessibility tree snapshots for inspection and document archiving.

Key facts

• https://github.com/descoped/mcp-web-scraper
• Browser & Web Automation, Web Search & Research
• automation, caching, playwright, web scraping

Part of MCP Servers

What can MCP Web Scraper do?

MCP Web Scraper provides 29 browser automation and extraction tools. It handles navigation, form interactions, tab management, drag-and-drop operations, network monitoring, and full-page screenshots. It also features automatic cookie banner resolution across dozens of frameworks and extracts structured article content directly into Markdown, plain text, or HTML.

How do I install MCP Web Scraper?

You can install and run the server using Docker by executing docker run -p 3001:3001 descoped/mcp-web-scraper. Alternatively, you can clone the repository from GitHub, install dependencies with npm install, compile the TypeScript code with npm run build, and launch the service with npm start.

Is MCP Web Scraper open source?

Yes, MCP Web Scraper is open source software published under the MIT license. You can inspect its source code, modify it, or deploy it on your own infrastructure via its GitHub repository.

Which MCP clients work with MCP Web Scraper?

Any MCP client that supports connecting to HTTP or SSE server endpoints can interact with MCP Web Scraper. This includes custom Node.js and TypeScript MCP client implementations using the official SDK, as well as desktop clients configured for web-based MCP endpoints.

How does MCP Web Scraper handle cookie consent popups?

The server includes built-in consent handlers supporting more than 25 common consent frameworks and over 30 languages. It automatically identifies and dismisses standard consent dialogs prior to extracting data or capturing page screenshots.