Bots and Scraping
Separating automated traffic from real users, then deciding what to do with each.
The part of scraping that is most often underestimated is how much of the traffic you believe is human is actually scripted. These articles cover identification (the real difference between rule matching and machine-learned scoring), response (what challenges, rate limiting and blocking each cost you), and one boundary that is ignored repeatedly: business-level abuse — bulk signups, promotion farming — is decided by data only you hold. Edge signals are an input, not the decision.
What we conclude in this topic
- A WAF matches patterns, bot management classifies requests — you need both.
- Business-rule abuse (promotion farming, bulk registration) is decided by your data, not by the edge.
- Verified crawlers must be handled explicitly, or you will lock out search engines along with the scrapers.
- Turnstile raises the cost of automation; rate limiting answers a different question. Keep both.
- Measure first: most teams underestimate the true share of automated traffic.
All articles (5)
Why WAF Rules Miss Bots — and Where Bot Management Starts
A WAF asks if a payload matches a known attack. Bot control asks if the caller behaves like a user. Tiers differ in what you can write rules against.
How to Prevent Malicious Web Scraping
Scraping is not one problem: search crawlers, competitor price monitoring and bulk extraction need opposite responses. How to tell them apart.
Preventing Automated Signups and Free-Tier Abuse in AI Apps
Automated signups poison user data and the inference bill at once. Six defence layers, from email reputation to invisible verification and quotas.
How to Protect AI Video Platforms from Bots and Scrapers
Video hotlinking, catalogue scraping and fake engagement drain CDN budget. Signed URLs, player endpoints, bot fingerprinting and cost governance.
How to Distinguish Legitimate Users from Malicious Bots
Bots are not automatically bad. Evidence across credential, environment and behaviour layers, real bot score semantics and a response ladder.
Related Cloudflare capabilities
Bot Management
Classifies automated traffic with a per-request score, separating scripts and scrapers from real users without a static rule for each one.
Turnstile
A CAPTCHA replacement that verifies visitors without making them solve a puzzle, used on signup, login and form endpoints.
WAF
Managed and custom rules that inspect HTTP requests at the edge, blocking common attacks before they reach your origin.
Frequently asked questions
If we already run a WAF, do we need bot management?
Yes — they solve different problems. A WAF matches requests against rules: this pattern, this action. Bot management scores the request itself, which is the only workable answer to automation that changes faster than rules can be written. In practice they are deployed together, with the score feeding the rule.
Can bot management stop promo farming?
It can stop the automated part: scripted signups, disposable infrastructure, credential stuffing. Whether a specific human is abusing a promotion depends on account history, funding relationships and campaign rules — data you hold. Edge signals are an input to that decision, not the decision itself.
Will blocking break search engine crawlers?
It can. Blocking purely by score band, without handling the verified crawlers you depend on, is how teams discover this the hard way. Cloudflare verifies major crawlers, and your rules should treat verified bots explicitly. Checking that list before enabling blocking costs almost nothing.
Other topics
- API and AI Security — Protecting the endpoints that actually cost money: contract validation, identity, quotas and automation control.
- Global Network and DDoS — Latency, availability and attack absorption for users who are far from your origin.
- Selection and Procurement — Comparing options, checking capability boundaries, and buying Enterprise without guessing.