The best web scraping tools in 2026 are Zyte, Bright Data, Apify and Oxylabs for production data pipelines, with Firecrawl as the pick for teams feeding web content into large language models, and Scrapy, Playwright and Crawl4AI as the leading free, open-source options. We scored eight commercial tools on one published, weighted framework, checked every price on the vendor's own pricing page in November 2026, cross-checked unblocking against the largest independent benchmark of scraping APIs, and added the criterion most comparisons skip: how well each tool fits a company that has to comply with the GDPR.
In short
- Best overall: Zyte (94.5/100), pay-as-you-go from $0.13 per 1,000 HTTP responses (from $0.06 on monthly commitments), with $5 of free credit for 30 days (Zyte pricing).
- Best EU vendor with a large proxy network: Oxylabs (88.5/100), from $30/month, ISO/IEC 27001 and SOC 2 Type II, Lithuanian entity (Oxylabs pricing).
- Best for hard targets at scale: Bright Data (87.5/100), $1.50 per 1,000 parsed records pay-as-you-go and 5,000 free records a month (Bright Data pricing).
- Best platform for business teams: Apify (87.5/100), free plan with $5 of monthly usage and paid plans from $19/month, run by a Czech company (Apify pricing).
- Best AI web scraper for RAG pipelines: Firecrawl (79/100), 1,000 free credits a month and LLM-ready Markdown by default, but weak on protected sites (Firecrawl pricing).
- Best free and open source: Scrapy for crawling at scale, Playwright or Selenium for JavaScript-heavy sites, Crawl4AI for LLM-ready Markdown.
- The legal point no tool solves for you: France's CNIL says scraping "is not prohibited per se, but must be analysed on a case-by-case basis" (CNIL). You remain the data controller, whichever vendor you pick.
A web scraping tool is software that automatically requests web pages, renders them when needed, and turns their content into structured data such as JSON, CSV or Markdown. In 2026 the market has four groups: web scraping APIs that handle proxies, browsers and blocking for you (Zyte, Bright Data, Oxylabs, ScrapingBee, ScraperAPI); platforms where you run and schedule ready-made scrapers (Apify); no-code tools for non-developers (Octoparse, Browse AI); and AI web scrapers that return clean text for language models (Firecrawl, Crawl4AI). Underneath many of them sit free, open-source libraries such as Scrapy, Playwright, Selenium and Beautiful Soup.
The reason the category matters more each year is defensive pressure. Imperva's 2025 Bad Bot Report found that automated traffic reached 51% of all web traffic in 2024, the first time in a decade that bots surpassed humans, and that malicious bots alone made up 37% (Thales, April 2025). With that much hostile automation, a growing share of high-value sites (retail, travel, classifieds, search) invest in bot detection, so unblocking is usually the first thing you pay a vendor for. The second driver is AI: agents and RAG systems now consume web pages directly, which is why output format has become a buying criterion in its own right.
What Changed in Web Scraping Since 2025?
Since 2025, scraping has become harder on Cloudflare-protected sites, more consolidated among a few vendor groups, and increasingly driven by AI agents through MCP servers. Four dated changes shape tool choice in 2026:
- July 2025, Cloudflare flipped the default. Cloudflare announced it would block AI crawlers by default unless they pay or are allowed, and launched a "pay per crawl" model for publishers (Cloudflare, 1 July 2025). Crawling sites behind Cloudflare now depends more on the site owner's explicit choice, and if the data will train or ground an AI model, the EU AI Act adds its own documentation duties (see our GDPR and AI Act compliance guide).
- June 2025, consolidation. Lithuania's Oxylabs group acquired the French API ScrapingBee, which keeps operating under its own brand (Tech.eu, 20 June 2025).
- 2025-2026, MCP servers everywhere. Zyte, Oxylabs and Apify ship Model Context Protocol servers so that AI agents such as Claude or ChatGPT can call their scrapers directly (Zyte, Oxylabs, Apify).
- June 2026, more audited compliance. Bright Data published a PwC ISAE 3000 Type II assurance report on its compliance and ethics controls, dated 30 June 2026 (Bright Data).
We scored each tool from 1 to 5 on five weighted criteria and converted the result to a score out of 100 (a tool with 5 on everything gets 100). Scores rest on features and prices published on official vendor pages, verified on 2 November 2026, plus one external measurement. We did not run our own lab benchmark; the testing section below explains how to measure success on your own targets. Every sub-score follows a written rule, listed under the comparison table.
Only 4 of 11 scraping APIs opened protected sites more than 80% of the time in Proxyway's December 2025 benchmark: Zyte, Decodo, Oxylabs and ScrapingBee (Proxyway, 4 December 2025). The test ran each API against 15 protected sites. Zyte led on both success and speed, Firecrawl placed last, and Apify was assessed separately because of its marketplace model. Bright Data was not included, so its unblocking score is capped below the top mark.
| Criterion |
Weight |
5 means |
3 means |
| Unblocking (documented features + Proxyway 2025 result) |
25% |
Proxies, JS rendering and CAPTCHA handling included, billed only on success, and the top result in the independent test |
Anti-blocking partly sold as paid add-ons, or a weak independent result |
| Output & AI-readiness |
20% |
Structured JSON and LLM-ready Markdown, AI extraction, MCP or agent access |
Raw HTML or CSV, extraction you build yourself |
| Scale & reliability |
15% |
High concurrency, scheduling, published uptime, enterprise contracts |
Limited concurrency, no published SLA |
| Compliance & EU fit |
20% |
Audited or certified programme, GDPR data processing agreement, EU contracting entity |
Basic terms and privacy policy, non-EU entity, little published on compliance |
| Value & access |
20% |
Free tier or credit, self-serve pricing, low cost per 1,000 pages |
Annual-only or opaque pricing, no free access |
Compliance gets 20% because, for a European company, it is where scraping projects actually fail. The Dutch data protection authority (Autoriteit Persoonsgegevens) concluded in its May 2024 guidance that scraping by private organisations and individuals is almost never permitted under the GDPR, unless it is done in a very targeted way (Autoriteit Persoonsgegevens, May 2024). A tool cannot make your project legal, but a vendor with an audited programme, an EU entity and a data processing agreement makes your own assessment easier to defend.
Zyte leads our 2026 scorecard with 94.5/100, followed by Oxylabs (88.5), Bright Data and Apify (87.5 each) and ScrapingBee (83). Firecrawl (79) is still the best fit for AI use cases: it has the best output score, but unblocking carries 25% of the weight and it placed last in the independent test.
| Tool |
Entry price (verified 2 Nov 2026) |
Free access |
Unblock |
Output |
Scale |
Compliance |
Value |
Score /100 |
| Zyte |
From $0.13/1K HTTP responses pay-as-you-go; from $0.06/1K on commitment |
$5 credit, 30 days |
5 |
4.5 |
4.5 |
5 |
4.5 |
94.5 |
| Oxylabs |
$30/mo Starter |
$5 welcome credit |
4.5 |
4 |
4 |
5 |
4.5 |
88.5 |
| Bright Data |
$1.50/1K records pay-as-you-go; $499/mo Scale |
5,000 records/mo |
4.5 |
4.5 |
5 |
4.5 |
3.5 |
87.5 |
| Apify |
$19/mo Starter ($17/mo billed annually) |
Free plan, $5/mo usage |
4 |
4.5 |
4.5 |
4.5 |
4.5 |
87.5 |
| ScrapingBee |
$19/mo Hobby (75,000 credits) |
1,000 credits |
4.5 |
4 |
3.5 |
4 |
4.5 |
83 |
| Firecrawl |
$19/mo Hobby ($16/mo billed annually) |
1,000 credits/mo |
3 |
5 |
4 |
3.5 |
4.5 |
79 |
| ScraperAPI |
$49/mo Hobby (100,000 credits) |
1,000 credits/mo free plan + 7-day trial |
4 |
3.5 |
4 |
3 |
4 |
74 |
| Octoparse |
From $69/mo Standard |
Free plan, 10 tasks |
3 |
3.5 |
3 |
3 |
4 |
66 |
Score = (Unblock × 25% + Output × 20% + Scale × 15% + Compliance × 20% + Value × 20%) ÷ 5 × 100. Prices verified on 2 November 2026 on each vendor's official pricing page; they exclude VAT and change often. Credit-based plans are not directly comparable: one "credit" can mean one simple page or a fraction of a rendered one.
How each sub-score was assigned. Unblock: 5 = proxies, rendering and CAPTCHA handling included and the top result in Proxyway 2025 (Zyte); 4.5 = the same stack plus more than 80% success in Proxyway (Oxylabs, ScrapingBee), or the full stack documented but never independently tested (Bright Data, capped for that reason); 4 = stack included but outside the 80% group (ScraperAPI) or results that depend on the individual scraper (Apify); 3 = last place in Proxyway (Firecrawl) or residential proxies and CAPTCHA solving billed as add-ons (Octoparse). Output: 5 = LLM-ready Markdown by default plus JSON extraction (Firecrawl); 4.5 = structured or parsed data plus MCP access (Zyte, Bright Data, Apify); 4 = AI parsing at extra cost or limited (Oxylabs, ScrapingBee); 3.5 = HTML, CSV or JSON without an LLM-ready format (ScraperAPI, Octoparse). Scale: 5 = unlimited concurrency documented plus enterprise contracts (Bright Data); 4.5 = hosted scheduling and cloud runs plus enterprise plans (Zyte with Scrapy Cloud, Apify); 4 = high documented concurrency or published uptime, no hosted scheduler (Oxylabs, ScraperAPI, Firecrawl); 3.5 = concurrency capped by plan (ScrapingBee); 3 = 3 concurrent cloud processes on the entry paid plan (Octoparse). Compliance: 5 = security certification, GDPR data processing agreement and EU contracting entity (Zyte, Oxylabs); 4.5 = an audited compliance programme with a non-EU entity (Bright Data) or an EU entity with documented GDPR terms (Apify); 4 = EU entity, less published on certification (ScrapingBee); 3.5 = SOC 2 and a DPA but no EU entity or EU data residency (Firecrawl); 3 = non-EU entity and little published on compliance (ScraperAPI, Octoparse). Value: 4.5 = free tier or credit plus self-serve entry at $30/month or less; 4 = entry plan at $49-$69/month; 3.5 = the highest unit price in the list (Bright Data).
1. Zyte: best overall web scraping API (94.5/100)
Zyte is the best overall web scraping API in 2026: it had the best result in the largest independent test, the lowest committed rate in this list and an Irish contracting entity.
- Best for: data engineering teams that want to own their crawler code. Zyte maintains Scrapy, the open-source Python framework, and hosts crawlers on Scrapy Cloud.
- Price: billed per successful response across five difficulty tiers. Pay-as-you-go runs from $0.13 to $1.27 per 1,000 HTTP responses and from $1.01 to $16.08 per 1,000 browser-rendered responses, falling to "from $0.06 per 1,000 successful responses" on $100, $200 and $500 monthly commitments (Zyte pricing).
- Free access: $5 of credit for 30 days.
- Independent test: first on success rate and fastest in Proxyway's December 2025 benchmark of 11 APIs.
- Compliance: ISO 27001 certified, with a GDPR data processing agreement (Zyte Trust Center).
- EU entity: Zyte Group Limited, Cork, Ireland (Zyte terms).
- Weakness: the tier grid makes costs hard to forecast. A rendered page in the top tier costs over 100 times a simple page in the bottom one ($16.08 vs $0.13 per 1,000), so price your real targets before you commit.
2. Oxylabs: best proxy-backed API from an EU vendor (88.5/100)
Oxylabs is the best choice when you need a large proxy network and a fully documented EU compliance file in one contract.
- Best for: teams scraping protected sites at volume from Europe.
- Product: now sold as Web API (/Scrape endpoint), backed by what Oxylabs describes as more than 177 million proxies in 195 countries, with 99.9% API uptime (Oxylabs).
- Price: plans start at $30/month, billed per successful result. The headline row (Web API /Scrape) costs $0.50 per 1,000 results on Starter, down to $0.25 on Enterprise. Custom domains cost $1.00 per 1,000 without JavaScript rendering ($0.75 on Enterprise) and $1.50 with rendering ($1.22), so check which row your target falls in (Oxylabs pricing).
- Free access: a $5 welcome credit.
- Independent test: above 80% success in Proxyway 2025, and among the fastest after Zyte.
- Compliance: ISO/IEC 27001:2022 certification covering its proxies and Web API, and a SOC 2 Type II report on its scraping endpoints (Trust Center).
- EU entity: oxylabs, UAB, Vilnius, Lithuania (Oxylabs terms).
- Weakness: AI parsing costs extra at $3.00 per 1,000 parsed pages, which is why it scores 4 on output.
3. Bright Data: best for hard targets at scale (87.5/100)
Bright Data is the strongest choice for heavily protected targets at high volume, and the only vendor here with a published third-party (PwC ISAE 3000) audit of its compliance and ethics controls.
- Best for: protected e-commerce, travel and search targets.
- Price: $1.50 per 1,000 records pay-as-you-go, billed only for delivered data, or $499/month for 384,000 records with extra records at $1.30 per 1,000. Residential proxies, JavaScript rendering, CAPTCHA solving and unlimited concurrency are included (Bright Data pricing).
- Free access: 5,000 records a month.
- Independent test: not part of Proxyway 2025, so its unblocking score is capped at 4.5 and rests on documented features.
- Compliance: customer vetting described in its Trust Center and a PwC ISAE 3000 Type II report dated 30 June 2026.
- EU entity: none by default; the contracting party is Bright Data Ltd. in Netanya, Israel, under Israeli law (Bright Data terms).
- Weakness: price. A record is parsed, structured data, not a raw page, so it is not like-for-like with Zyte's $0.13 per raw response; if you only need simple pages, you pay for parsing you do not use.
Apify is the best platform for teams that want ready-made scrapers they can schedule without writing code, tied on points with Bright Data.
- Best for: operations and growth teams.
- Product: thousands of scrapers ("Actors") in the Apify Store, scheduled from a web console, plus a hosted MCP server that lets Claude, ChatGPT or Cursor run them (Apify). Developers get Crawlee, an open-source SDK.
- Price: Starter $19/month ($17 billed annually), Scale $199 ($179), Business $999 ($899), with compute from $0.20 down to $0.13 per compute unit (Apify pricing).
- Free access: $5 of platform usage a month.
- Independent test: Proxyway assessed Apify separately and results varied by Actor: a G2 run returned 5,926 results in under 15 minutes, while a Hyatt run returned none.
- EU entity: Apify Technologies s.r.o., Prague, Czech Republic (Apify terms).
- Weakness: quality control. Actors are built by many developers and many add their own per-result price, so test reliability and real cost before you build a process on one.
5. ScrapingBee: simplest API for developers (83/100)
ScrapingBee is the simplest web scraping API for developers who want one endpoint and a predictable monthly bill.
- Best for: small development teams.
- Product: headless Chrome rendering and proxy rotation behind one API call, plus natural-language extraction with "AI Query" (ScrapingBee).
- Price: Hobby $19/month for 75,000 credits, Freelance $49 for 250,000, Startup $99 for 1,000,000 and Business $249 for 3,000,000 (ScrapingBee pricing). That is about $0.10 per 1,000 basic requests on Startup, the lowest per-request rate of any fixed monthly plan here; only Zyte's committed rate (from $0.06) is lower.
- Free access: 1,000 credits.
- Independent test: above 80% success in Proxyway 2025.
- EU entity: VostokInc SAS, Paris, France, under French law (ScrapingBee terms); part of the Oxylabs group since June 2025.
- Weakness: credit multipliers. JavaScript rendering costs 5 credits and premium proxies 10 or 25, and concurrency is capped by plan (25 requests on Hobby).
6. Firecrawl: best AI web scraper for RAG and LLM pipelines (79/100)
Firecrawl is the best AI web scraper for RAG and agent pipelines because it returns clean, LLM-ready Markdown by default, but it is the weakest tool here on protected sites.
- Best for: AI teams building RAG, agents and knowledge bases on mostly unprotected sources.
- Product: "LLM-optimized Markdown output by default", with search, scrape, crawl, site mapping, document parsing and change monitoring (Firecrawl). The core is open source under AGPL-3.0, so you can self-host it (GitHub).
- Price: Hobby $19/month ($16 billed annually) for 5,000 credits, Standard $99 ($83) for 100,000; 1 credit per page, plus 4 for JSON extraction (Firecrawl pricing).
- Free access: 1,000 credits a month.
- Independent test: placed last of the 11 APIs in Proxyway 2025. It fails fast instead of retrying internally, which Proxyway says "offloads the burden of retrying to the user", and it is "best suited for crawling the long-tail rather than individual protected targets".
- Compliance: "SOC 2 Type II compliant (renewal in progress)", with a DPA and zero data retention on Enterprise. Data is processed in the United States and Firecrawl "does not offer EU data residency" (Firecrawl security).
- Weakness: protected targets and EU data residency.
7. ScraperAPI: solid volume API (74/100)
ScraperAPI is a solid volume API for steady, high-throughput jobs on moderately protected sites.
- Best for: developers running large, regular jobs where speed matters more than compliance paperwork.
- Price: Hobby $49/month for 100,000 credits (20 threads), Startup $149 for 1,000,000 and Business $299 for 3,000,000, with 10% off on annual billing (ScraperAPI pricing). JavaScript rendering, premium proxies and CAPTCHA handling are included on all plans.
- Free access: a permanent plan with 1,000 credits a month, plus a 7-day trial with 5,000.
- Independent test: Proxyway called it a "strong all-rounder" and one of the fastest after Zyte, but it was not in the 80% group.
- EU entity: none; the contracting party is ScraperAPI LLC under Nevada law (ScraperAPI terms).
- Weakness: little published on compliance and no LLM-ready output, so an EU buyer must document the international transfer itself.
8. Octoparse: no-code desktop scraping (66/100)
Octoparse is the most accessible option for a non-developer who needs a one-off dataset.
- Best for: marketing or sales analysts who prefer point-and-click to code.
- Price: a free plan with 10 tasks and 50,000 exported rows a month; Standard from $69/month for 100 tasks and 3 concurrent cloud processes (16% less on annual billing); Professional $249/month (Octoparse pricing).
- Anti-blocking: IP rotation is included on Standard. Residential proxies and automatic CAPTCHA solving are available, but billed from prepaid add-on credit: $3 per GB and $1-1.5 per 1,000 CAPTCHAs.
- EU entity: none; Octopus Data Inc., Irvine, California, under Californian law (Octoparse terms).
- Weakness: desktop-centric workflows, low cloud concurrency on the entry plan and no LLM-ready output. Browse AI is the main no-code alternative, with 50 free credits a month and paid plans from $19/month billed annually (Browse AI pricing).
- Decodo (formerly Smartproxy, Lithuania): one of the four APIs above 80% in Proxyway 2025, with a free tier of up to 10,000 scraped pages and plans from $19/month (Decodo pricing). Not scored in this edition; on unblocking and price it competes directly with Oxylabs, so test both if you want a Lithuanian vendor.
- ZenRows: a developer API with 5,000 free credits a month and plans from $19/month (ZenRows pricing); in Proxyway's test it suffered most when the request rate rose to 10 per second.
- Scrapfly, Diffbot, ParseHub and Jina Reader: respectively a developer API close to ScrapingBee, an AI extraction and Knowledge Graph service closer to a data provider, a desktop no-code scraper like Octoparse, and a URL-to-Markdown endpoint for LLM prototypes without crawling or unblocking.
The best free web scraping tools are open-source libraries: Scrapy for large crawls, Playwright, Puppeteer or Selenium for JavaScript-heavy sites, Beautiful Soup for parsing HTML, and Crawl4AI or ScrapeGraphAI when the output feeds an LLM. They cost nothing in licences, but you supply the proxies, servers and anti-blocking work yourself.
| Library |
Language |
Licence |
Best use |
| Scrapy |
Python |
BSD-3-Clause |
Large, scheduled crawls with pipelines and retries |
| Playwright |
TypeScript, Python, Java, .NET |
Apache-2.0 |
Rendering JavaScript-heavy pages, logins, clicks |
| Puppeteer |
Node.js |
Apache-2.0 |
Headless Chrome automation in JavaScript projects |
| Selenium |
Java, Python, C#, JavaScript, Ruby |
Apache-2.0 |
Browser automation for teams that already use it for testing |
| Beautiful Soup |
Python |
MIT |
Parsing HTML you already downloaded |
| Crawl4AI |
Python |
Apache-2.0 |
Turning sites into LLM-ready Markdown |
| ScrapeGraphAI |
Python |
MIT |
Extracting fields with an LLM prompt instead of selectors |
Licences were checked on each project's GitHub repository, and star counts were read from the GitHub API on 2 November 2026: Playwright about 99,000, Puppeteer about 96,000, Crawl4AI about 87,000, Scrapy about 65,000, Selenium about 35,000 and ScrapeGraphAI about 32,000 (Firecrawl's open-source core, covered above, has about 198,000). Stars measure attention, not fitness for production. Free commercial tiers are the other route: ScraperAPI (1,000 credits a month), Firecrawl (1,000 credits), Bright Data (5,000 records), Apify ($5 of usage) and Octoparse (10 tasks) all have permanent free plans.
Library or API? Use a library when your targets are unprotected, volumes are modest and you have a Python or JavaScript developer to maintain selectors. Move to an API when you lose more engineering time to blocks, CAPTCHAs and proxy management than the API would cost; a common pattern is Scrapy for crawl logic with Zyte or another API as the download layer.
For a JavaScript-heavy page, a few lines of Python with Playwright are enough to start (install with pip install playwright and playwright install chromium, then replace the URL and CSS selector with your own):
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
page = p.chromium.launch().new_page()
page.goto("https://example.com/products")
print(page.locator("h2.product-title").all_inner_texts())
Web Scraping API vs No-Code vs AI Web Scraper: Which Should You Choose?
Choose by the job and the team, not by the longest feature list. The table below maps the most common profiles to the tools that scored best for them.
| If you are... |
Choose |
Why |
| A data engineer with Python skills |
Zyte (+ Scrapy) |
Low pay-as-you-go rate, EU contract, you own the crawler code |
| Scraping protected e-commerce, travel or search at volume |
Bright Data or Oxylabs |
Deepest proxy networks, billing on success |
| An operations or sales team without developers |
Apify Store, Octoparse or Browse AI |
Ready-made scrapers and visual builders |
| Building RAG, agents or a knowledge base |
Firecrawl or Crawl4AI |
LLM-ready Markdown by default; see our AI agents service |
| A developer wanting a cheap, simple API |
ScrapingBee or ScraperAPI |
One endpoint, monthly plans from $19 |
| On a zero budget with unprotected targets |
Scrapy, Playwright, Beautiful Soup |
Free and open source |
Scraping is rarely the end goal. The data usually feeds a CRM, an ERP, a price engine or a dashboard, which is where robotic process automation (RPA) and integration work take over; our practical RPA guide for SMEs and our comparison of workflow automation platforms cover that side. For sales teams, our article on how AI agents automate B2B sales shows where scraped company data fits into prospecting.
Test a web scraping tool on your own target sites for a week before committing, because success rates depend more on the target than on the vendor. Our scores rank documented features; only your own run measures how a tool performs on the pages you need. A simple protocol:
- Pick 3 types of target: a static page, a JavaScript-heavy page and a page behind bot protection, with 100 URLs each.
- Run the same list through 2-3 shortlisted tools using their free credits or trials.
- Record three numbers per target type: success rate (valid content returned, not just HTTP 200), median response time, and cost per 1,000 successful pages including credit multipliers.
- Check the output: is it the field you need, in a format your pipeline can load without manual cleaning?
- Log your legal checks next to the results: collection criteria, robots.txt status and data categories excluded.
Free credits from Zyte ($5), Oxylabs ($5), ScrapingBee (1,000), ScraperAPI (5,000 trial) and Firecrawl (1,000) are enough to run this test at no cost. For a reference point, Proxyway's December 2025 test found overall success rates ranging from about 22% on Shein to about 98% on Zillow across the APIs it tested (Proxyway), so the same tool can look excellent or poor depending on the target.
Is Web Scraping Legal in the EU Under the GDPR?
Web scraping is not illegal in the EU, but scraping personal data requires a legal basis under the GDPR, and in practice that basis is usually legitimate interest under Article 6(1)(f), which allows processing necessary for the legitimate interests of the controller or a third party only when people's rights do not prevail (GDPR, Regulation (EU) 2016/679). Public availability is not permission: the Dutch authority's guidance says scraping is almost never allowed for private parties and gives the example of scraping social media profiles to decide whether someone gets an insurance policy (Autoriteit Persoonsgegevens).
The CNIL's how-to sheet was written for scraping used to develop AI systems, but it is the most concrete EU guidance available for any scraping project. It separates what is required from what strengthens your case. Mandatory measures:
- Define collection criteria in advance and filter out data categories you do not need.
- Exclude sites that by their nature contain sensitive data, such as platforms used mainly by minors.
- Delete immediately any irrelevant data collected despite those criteria.
- Exclude sites that clearly object to scraping, for example through robots.txt or CAPTCHAs.
Additional safeguards that reinforce legitimate interest: default exclusion lists of intrusive-content sites (health forums, adult sites, genealogy databases), limiting collection to freely accessible data, wide publicity about the collection, a discretionary prior right to object, anonymisation or pseudonymisation right after collection, and no combining of data on individual identifiers (CNIL).
Copyright is a separate test. Article 4 of the Digital Single Market Directive allows text and data mining of lawfully accessible content, but rights holders can reserve that use, for content online, through machine-readable means (Directive (EU) 2019/790). Robots.txt is the most common signal, although the standard itself says its rules "are not a form of access authorization" (RFC 9309).
"Most scraping projects fail on the second question, not the first," says Alfons Marques, founder of Technova Partners. "Every vendor can get you the page. What decides success is what you are allowed to keep, and which system will use the data on Monday morning." If your project touches personal data, involve your data protection officer before the first crawl; our guide to the DPO role and our GDPR and AI Act compliance guide explain how to document legitimate interest.
Web scraping tools cost from $0 to about $70 a month at entry level in 2026, while scale plans run from $99 to $999 a month or more; per 1,000 simple pages, prices range from about $0.06 to $1.50. Rendered or heavily protected pages cost several times more on every platform.
Cost per 1,000 pages is the fairest comparison, keeping in mind that units differ:
- ScrapingBee: about $0.10 per 1,000 basic requests on Startup ($99 / 1,000,000 credits); about $0.50 with JavaScript rendering
- Zyte: $0.13-$1.27 per 1,000 HTTP responses, $1.01-$16.08 rendered, pay-as-you-go with no subscription; from $0.06 on monthly commitments
- ScraperAPI: about $0.15 per 1,000 credits on Startup ($149 / 1,000,000)
- Firecrawl: about $0.99 per 1,000 pages on Standard ($99 / 100,000 credits)
- Oxylabs: $1.00 per 1,000 results on custom websites ($0.75 on Enterprise), $1.50 with JavaScript rendering on Starter
- Bright Data: $1.50 per 1,000 parsed records pay-as-you-go, $1.30 for extra records on Scale
Apify bills compute rather than pages, and many Store Actors add their own per-result price, so estimate with a test run. Open-source libraries have no licence cost, but proxies and servers do. Credit multipliers matter more than headline prices: on ScrapingBee, a premium-proxy request with rendering costs 25 credits, which turns about $0.10 per thousand pages into about $2.50.
If you would like a second opinion on a compliant data collection pipeline, our data strategy consulting team can help design it, and our data analytics services cover the dashboards and models the scraped data ends up feeding.
Frequently Asked Questions
Zyte scored highest in our comparison at 94.5/100: pay-as-you-go from $0.13 per 1,000 HTTP responses, the best result in Proxyway's December 2025 independent test, ISO 27001 and an Irish legal entity. Oxylabs (88.5) is the best EU vendor with a large proxy network, Bright Data (87.5) leads on hard targets at scale, Apify (87.5) on ready-made scrapers for business teams, and Firecrawl (79) on LLM-ready output.
Is web scraping legal?
It depends on the data and the jurisdiction. Scraping public, non-personal data is generally lawful in both the EU and the US, but contracts, copyright and privacy law still apply. In the EU, personal data needs a GDPR legal basis, usually legitimate interest, plus safeguards; the CNIL says each case must be analysed on its own, and the Dutch authority considers most private scraping of personal data unlawful. In the US, the Ninth Circuit found serious questions over whether scraping public profiles breaches the CFAA in hiQ v. LinkedIn (9th Cir., April 2022), yet hiQ later lost on breach of LinkedIn's User Agreement and a December 2022 consent judgment ordered it to pay $500,000 (National Law Review). In January 2024 a federal court ruled that Meta's terms did not bar Bright Data's logged-out scraping of public data (N.D. Cal.).
What is the difference between web scraping and web crawling?
Web crawling discovers pages by following links and building a list of URLs, which is what search engines do. Web scraping extracts specific fields, such as prices, names or text, from those pages into structured data. Most business projects do both: a crawler finds the product or listing pages, and a scraper turns each one into a row in your dataset.
Is Scrapy or Beautiful Soup better?
Scrapy is better for crawling many pages, because it handles requests, retries, scheduling and export pipelines; Beautiful Soup only parses HTML you have already downloaded. Use Beautiful Soup with the requests library for a quick script on a few pages, and Scrapy once you crawl whole sites on a schedule.
Python has the strongest scraping ecosystem: Requests plus Beautiful Soup for quick scripts, Scrapy for large scheduled crawls, Playwright or Selenium for JavaScript-heavy pages, and Crawl4AI or ScrapeGraphAI when the output feeds an LLM. Zyte, Oxylabs, ScrapingBee and ScraperAPI all publish Python examples, so you can keep Scrapy for crawl logic and use an API as the download layer.
Can ChatGPT or AI do web scraping?
ChatGPT and other AI assistants can read individual pages when browsing is enabled, but they are not bulk scrapers: they do not schedule crawls, rotate proxies or return thousands of records. To scrape at scale with AI, use an AI web scraper such as Firecrawl or Crawl4AI, or connect the assistant to a scraping service through an MCP server, such as those of Apify, Zyte or Oxylabs.
For non-developers, Octoparse and Browse AI are the easiest, because you click on the data you want instead of writing code. Apify Store Actors are a close second. For beginners who can write a little Python, Beautiful Soup is the gentlest start, moving to Scrapy as projects grow.
Yes. Scrapy, Playwright, Beautiful Soup and Crawl4AI are free and open source, and ScraperAPI, Firecrawl, Bright Data, Apify and Octoparse all keep permanent free plans. Free tiers are enough to validate a target site; production needs a paid plan or your own infrastructure.
Is there a web scraping Chrome extension?
Yes. Web Scraper (webscraper.io) offers a free browser extension for Chrome and Firefox that runs locally, with a cloud version from $50/month billed annually for scheduled jobs (Web Scraper pricing). Extensions suit one-off lists of a few hundred pages. Because they run in your own logged-in browser, check the site's terms before scraping anything behind a login.
Is Selenium or Playwright better for scraping?
Playwright is usually the better choice for new scraping projects: it waits automatically for elements, drives Chromium, Firefox and WebKit through one API and supports Python, Node.js, Java and .NET. Selenium is the W3C WebDriver standard with the widest language support, so it makes sense when your team already runs Selenium test suites.
How do websites detect and block web scraping?
Websites detect scrapers through IP reputation and request rates, browser fingerprints such as headless-browser markers and unusual TLS or header patterns, behaviour such as perfectly regular timing, and challenges such as CAPTCHAs or JavaScript checks run by bot managers. That is why scraping APIs sell rotating residential proxies, real browsers and CAPTCHA handling. Blocking is common; legal action is rarer and usually concerns terms of use, copyright or personal data.
If you want to collect web data without creating a GDPR problem, talk to us: we can assess the legal basis with you, shortlist the right tool and connect the data to the systems that use it.
