By 
A year ago, Cloudflare declared what it called Content Independence Day. The pitch was simple: website owners should get to decide whether AI companies could crawl their content, and if they said yes, they should get paid for it. It was a good idea that landed at a moment when publishers everywhere were watching AI chatbots quietly absorb their work with nothing coming back.
On July 1, 2026, Cloudflare returned with year two of that push. The announcement introduced a new classification system for web crawlers, a dashboard that shows publishers exactly how AI bots are using their content, and an evolution of last year's Pay Per Crawl feature into something called Pay Per Use. The headline change is a September 15, 2026 deadline for so called "mixed use" crawlers, bots that blend search indexing, AI training, and AI agent activity into a single pass. After that date, any crawler that refuses to separate those three purposes will be blocked by default on every page that carries ads, for new customers and for existing free tier customers who haven't changed their settings.
On paper, this looks like real progress. Cloudflare is naming a problem that publishers have complained about for two years: Google's crawler, in particular, mixes search indexing with the data collection that powers AI Overviews and Gemini, and site owners have had no clean way to accept one without accepting the other. Cloudflare's own numbers make the imbalance concrete. According to its published data, Google's ratio of pages crawled to human visitors referred back sits at roughly 14 to 1. OpenAI's sits at roughly 1,700 to 1. Anthropic's sits at roughly 73,000 to 1. Whatever bargain used to exist between crawler and website, it broke down for AI a long time ago, and Cloudflare is right to say so.
But look closer at how Pay Per Use actually works, and a less flattering pattern shows up. This isn't a fix for the small publisher. It's a system that, as currently built, mostly works for the publisher who was already big enough to matter.
How Pay Per Use Actually Creates Winners
The mechanics matter here, so it's worth being precise about what changes and what doesn't.
Blocking a mixed use crawler does not automatically generate income. It removes the free option. Whether a publisher then gets paid depends entirely on whether the AI company crawling their site has signed up to a Pay Per Use partnership with Cloudflare. As of this announcement, exactly two such partners exist: Ceramic.ai and You.com. Both are AI search products looking to build content relationships, and both are considerably smaller than the AI companies most publishers actually worry about. Neither OpenAI, Google, Anthropic, nor Perplexity has committed to this payment model yet.
That gap matters more than it sounds like it should. A publisher who blocks mixed crawlers today is betting that the AI companies actually driving traffic and training value will eventually join a scheme that pays them. Until that happens, blocking a mixed crawler mostly means one thing: the publisher disappears from that crawler's index, gets no payment, and loses whatever search visibility that crawler used to provide.
Compare that to what's already happened in the licensing market that predates this announcement. Cloudflare notes that publishers have signed more than 50 major content licensing agreements over the past year. The names that keep coming up in coverage of these deals are the Associated Press, Time, The Atlantic, Reddit, and Getty Images. These are organizations with either enormous archives, legal departments built for negotiation, or both. When an AI company wants training data or citation rights badly enough to pay for it, it goes to the publisher whose content is irreplaceable or whose lawyers are expensive to ignore. It does not go looking for the mid sized regional tech blog with a loyal but modest readership, no matter how good that blog's reporting is.
Cloudflare has built genuine infrastructure here: a marketplace, a classification system, an attribution dashboard. What it hasn't built is a reason for AI companies to seek out small publishers within that marketplace. Infrastructure without a redistribution mechanism just formalizes whatever leverage already existed before the infrastructure was built.
The Small Publisher's Actual Dilemma
This is where the announcement's real cost lands, and it's worth stating plainly because Cloudflare's press language never quite does.
A smaller publisher now faces two doors, and neither one is good.
Door one: allow mixed crawlers. Stay discoverable in search and in AI Overviews. Keep whatever referral traffic still trickles back. Continue getting scraped for AI training and agent use with no compensation, exactly as before this announcement, because "allow" was always the default that cost publishers nothing to accept and everything to enforce against.
Door two: block mixed crawlers. Lose inclusion in Google Search and AI Overviews as a package deal, since Googlebot, Applebot, and Bing's crawler are all mixed use and will be treated according to the most restrictive rule a site sets. Wait, with no guarantee, for an AI company to decide your content is worth a direct licensing conversation. For most publishers outside the handful of major outlets already naming names in the coverage, that wait has no clear end date.
A publisher like ours sits squarely in that second, harder position. We don't have AP's thirty year archive or Reddit's user generated data trove to negotiate with. What we have is original reporting on Kenyan and African tech, written by people who know the space. That's valuable to readers. Whether it's valuable enough to earn a bespoke Pay Per Use deal with an AI company that has never named a partner our size is a genuinely open question, and right now, Cloudflare's own framework doesn't answer it.
This is the trade most coverage of the announcement has missed. The story isn't just "Google versus everyone else." It's "publishers with leverage versus publishers without it," and Cloudflare's new rules, however well intentioned, sit on top of that divide rather than closing it.
This Isn't Necessarily Malicious, Just Structurally Biased
It's worth being fair to Cloudflare here, because the alternative reading, that this is a deliberate scheme to help media giants at everyone else's expense, isn't quite right either.
Cloudflare didn't choose which publishers get licensing deals. The AI companies did. Cloudflare built a marketplace and a set of controls; it's the AI companies deciding who to approach with money on the table. That's not unusual. Every licensing market in every industry defaults toward whoever has leverage first, whether that's music labels negotiating streaming rates or film studios negotiating with theater chains. Scale and legal weight have always bought better terms than a good product alone.
What's different here is the stakes. Publishers aren't just losing negotiating leverage in an established market. They're watching the terms of a brand new market get set before most of them even have a seat at the table, and the September 15 deadline compresses that timeline further. Cloudflare says it will spend the next two months gathering feedback before finalizing the defaults, which is the right instinct. Whether that feedback period actually changes anything for smaller publishers, or just refines the rules that already favor whoever showed up with a licensing deal in hand, is the thing worth watching over the next ten weeks.
What Would Actually Fix This
If the goal is genuinely a fairer web, and not just a better negotiating position for outlets that already had one, a few things would need to exist that don't yet.
Collective bargaining for smaller publishers. Something closer to a syndication pool, where publishers below a certain scale can pool their content and negotiate as a bloc rather than individually. Getty built exactly this kind of leverage for photographers over decades. Nothing similar exists yet for independent digital publishers, though the demand clearly does.
A mandatory minimum rate rather than pure negotiation. If Pay Per Use payments are set entirely by one on one deals between AI companies and individual publishers, small publishers will always lose that negotiation before it starts. A baseline per citation rate, set either by Cloudflare or by regulation, would give every publisher something rather than nothing while bigger deals still exist above that floor for outlets with genuine leverage.
Regulatory pressure that doesn't wait for the market to sort itself out. The UK's move to force Google to let publishers opt out of AI search results without losing their regular search ranking is a useful precedent. It says, in effect, that publishers shouldn't have to accept a package deal just to stay visible. A similar principle applied to payment, some baseline compensation for citation regardless of publisher size, would do more for small publishers than any voluntary marketplace Cloudflare builds on its own.
None of these exist today. Cloudflare has built the pipes. Nobody has yet built the part that makes sure water reaches every house on the street, not just the ones closest to the source.
The Lock Without the Key
Cloudflare gave every publisher, big or small, a lock for their front door. That's real, and it's more than existed before. But a lock only matters if there's something on the other side worth protecting, and a system that pays out only where an AI company has already decided to show up with a signed deal isn't compensation. It's a waiting room.
For AP, Time, Reddit, and Getty, that waiting room has a door that opens. For most publishers, including outlets like ours, it's still just a room.
The real test of whether this announcement changes anything won't be how many crawlers get blocked on September 15. It will be how many AI companies sign Pay Per Use deals with publishers nobody's heard of, not just the ones everybody already has.
Comments