How to Prevent Malicious Web Scraping

Short answer: Scraping is not one problem: search crawlers you want, price monitoring you may tolerate, and bulk extraction you must block — and they need opposite responses, so classify before you block. On Cloudflare, Bot Fight Mode is free and only challenges known-bad clients; Super Bot Fight Mode (Pro) adds bot-score-based detection for definitely automated traffic; Enterprise Bot Management is the tier that lets you put bot score and JA3/JA4 fingerprints into your own rules. robots.txt declares intent but enforces nothing.

Three kinds of scraping that need opposite responses

Treating all automated traffic as hostile is the fastest way to damage your own distribution. Separate it first:

Kind	Who	What you want
Search and AI crawlers	Googlebot, Bingbot, and the AI crawlers feeding answer engines	Let them in — they are acquisition channels, especially for AI search visibility
Legitimate aggregation	Price comparison, partner integrations, monitoring	Allow, but rate-limit and watch
Bulk extraction	Competitors, data resellers, model trainers	Block, or make it uneconomic

The third category is where the work is, and the mistake that costs the most is applying third-category controls to first-category traffic. Rate limiting a verified crawler is a self-inflicted SEO problem.

Signals that separate bulk extraction from real traffic

• No asset requests. A scraper fetches HTML and skips CSS, JS and images. Real browsers do not.
• Depth without progression. Thousands of detail pages, no search, no add-to-cart, no session continuity.
• Machine-regular timing. Fixed intervals or perfectly parallel workers; humans are irregular.
• Concentrated identifiers. One ASN, one user agent, one fingerprint family.
• Parameter walking. Sequential IDs, cursor enumeration, or systematic filter combinations.

The first two are available to anyone with access to edge logs; the last two usually need bot signals rather than your own analytics.

The controls, in the order they should be applied

1. Declare intent in robots.txt — and understand what it does not do. Cloudflare’s own documentation is explicit: robots.txt compliance is voluntary, and it does not prevent crawlers at a technical level. Enforcement requires actual controls. Managed robots.txt is available on all plans, but Cloudflare only injects its version when you enable Bot Preference Sync and at least one policy is not set to Allow.

2. Decide an AI crawler policy explicitly. The AI bot controls let you set a posture per purpose — Search, Agent and Training — with three options each: block everywhere, block only on pages with ads, or allow; they are available on all plans including Free. The significance is strategic, not technical: allowing Search while blocking Training keeps you visible in AI answers while declining to donate your content to training sets. For newly onboarded domains the defaults are documented as blocking Training and Agent on ad-bearing pages while allowing Search.

3. Use the bot tier your plan includes. This is where most of the plan difference lives:

Tier	Detects	Control you get	Analytics
Bot Fight Mode (Free)	Simple bots from cloud hosting and headless browsers	One computationally expensive challenge, applied across the whole domain	Basic
Super Bot Fight Mode (Pro)	Simple bots and headless browsers	allow / block / challenge, static resource protection, JS detections. “Likely automated” is not available on Pro	Bot Report only
Business / Enterprise without BM	Many sophisticated bots	Adds Likely automated and definite automation	Dedicated Bot Analytics
Bot Management (Enterprise subscription)	Adds scoring and fingerprinting in rules	bot score, ja3_hash / ja4, detection IDs, bot tags, per-path and per-IP scoping	Full

Two details worth committing to memory. Bot score is 1–99: 1 is certainly automated, 99 is certainly human, and 0 means not evaluated — which the documentation states does not indicate the request is safe or human. And only Bot Management subscribers can use the score in custom rules; other plans work with the traffic groups instead. The official template uses score lt 30, noting that scores below 30 are commonly associated with bot traffic.

4. Rate-limit what remains. Rule counts are 1 / 2 / 5 / 100 by plan; on Free the expression is limited to path and verified-bot characteristics. Counters accumulate per data centre, so an absolute ceiling still belongs in your application.

5. Add friction on the paths worth defending. Turnstile is free, server-side validation is mandatory, and a token is valid for 300 seconds and single-use. Put it on search, export, login and registration — not on your landing pages.

6. Make extraction uneconomic in the application. Pagination caps, per-account quotas, watermarked or delayed data, and honeypot endpoints. AI Labyrinth takes this idea to the edge by injecting invisible nofollow trap links, but note it is not a blocking action — it only records AI Labyrinth Served/Crawls in Security Analytics, and links remain live for a while after you turn it off.

Defence in depth

Cloudflare capabilities vs. what your application must own

Item	What Cloudflare provides	What your application must do
Declared policy	Managed robots.txt; AI bot policies on all plans	Decide the posture per purpose and keep it current
Bot detection	Three tiers by plan; score and fingerprints from Bot Management	Choose actions per path; accept that low tiers are coarse
Rate control	Rate limiting rules, plan-limited	Per-account and per-key quotas; absolute ceilings
Human verification	Turnstile with mandatory server-side validation	Wire it into the flows that matter; never trust the client alone
Traps and telemetry	AI Labyrinth; Security Analytics records	Honeypot endpoints and pagination caps in the app
Data economics	—	Decide what is worth protecting and make extraction unprofitable

Boundaries and non-applicable cases

• robots.txt is not enforcement. Cloudflare states plainly that compliance is voluntary and the file does not prevent crawling at a technical level. It is a declaration, not a fence.
• Hotlink Protection does not stop scraping. The documentation says explicitly that it has no impact on crawling; it protects image assets from being embedded elsewhere.
• A score of 0 is not a clean bill of health. It means the request was not evaluated — internal traffic and requests handled earlier by redirect rules land there. Do not build “score is not low, therefore allow” logic.
• Protecting static resources can block good bots. Static resource protection is documented as potentially blocking mail clients and similar useful agents. Scope it, and check what broke afterwards.
• Rate limiting verified crawlers costs you distribution. Whatever you throttle is traffic that will not show up in search results. Exclude verified bots deliberately rather than by accident.
• A single global counter is an anti-pattern. Cloudflare’s Durable Objects guidance warns that one global object is typically an anti-pattern and recommends sharding by IP or user, and says to prefer WAF rate limiting rules when you do not need custom logic.

FAQ

Can I block scraping on the free plan? Partly. You get Bot Fight Mode, which issues an expensive challenge across the whole domain against simple bots and headless browsers, plus robots.txt and the AI bot policies. Sophisticated scrapers will not be stopped, and you cannot scope actions per path the way higher tiers allow.

Why can’t I write a rule using bot score? Because using cf.bot_management.score in custom rules requires a Bot Management subscription. On other plans you write rules against traffic groups — automated, likely automated, likely human, verified bot — which is coarser but still useful.

Will blocking AI crawlers hurt my visibility? It depends which purpose you block. Blocking Training protects your content from being absorbed into models; blocking Search removes you from AI answers. These are separate decisions with opposite business consequences, and the controls let you make them separately.

Is scraping illegal, so can we just send a notice? Sometimes, and notices do work against identifiable commercial actors. But enforcement is slow, jurisdiction-dependent and useless against anonymous operators. Technical controls change the economics immediately; legal action is a complement, not a substitute.

Related reading

• How to Choose an Enterprise Web Application Firewall
• How to Mitigate HTTP Flood and Layer 7 Attacks
• How to Prevent API Abuse in AI SaaS
• Bot management capabilities