Cloudflare flips the default: AI crawlers blocked unless they pay
From July 1 Cloudflare began defaulting new zones to blocking unverified AI crawlers — asking owners at signup rather than requiring an opt-out — and launched Pay Per Crawl in private beta, a marketplace that answers a bot with HTTP 402 Payment Required and a price in a crawler-price header, or 200 with a crawler-charged receipt when the bot accepts. Cloudflare says its network fronts about 20% of the web.
The company's own Radar numbers are the argument: it measured OpenAI's GPTBot at roughly 750 fetches per referral and Anthropic's crawler at roughly 30,000, against about 10 for the old Google search bargain. Cloudflare acts as merchant of record and says it will delist crawlers that spoof identity.

WHY IT MATTERS
For anyone running a RAG pipeline or a product that fetches third-party pages, this changes the data supply chain, not just a CDN setting: newly onboarded origins behind Cloudflare now return 402 or a challenge to unverified bots, so ingestion needs to declare identity and handle a payment handshake instead of retrying on 429. Budget per-fetch cost, cache hard, and treat first-party corpora as the only input you control.
