The Enclosure of the Open Web and the Open Internet Toll booth: What’s Behind Pay-By-Crawl
Cloudflare recently proposed a system where AI companies and crawlers would pay websites for the right to crawl their content, a move framed as “content independence day”, a response to growing concerns from creators and publishers about the unlicensed scraping of the web for AI training. But while the proposal might appear to support fair compensation, in reality it paves the way for something far more dangerous: the enclosure of the open web under the banner of copyright enforcement.
At its core, “Pay-By-Crawl” shifts power not away from AI companies, but toward Cloudflare, which positions itself as the broker between AI companies and online publishers. It’s a classic case of infrastructure stepping into an enforcement role, and it’s deeply troubling coming from a company that has historically fought against copyright overreach, particularly in the DNS space.
What makes this especially puzzling is that Cloudflare, a staunch defender of intermediary neutrality in contexts like DNS resolver privacy and free expression and open Internet (read their excellent blog about the open Internet and sanctions), now proposes a system that formalizes and monetizes surveillance of user content. It does so by treating website content as a proprietary asset to be licensed per scrape, regardless of who the real author is, or whether they consent to this monetization in the first place.
There’s no single, stable definition of “publisher” or “creator” online. A vast amount of online content is remixed, copied, or reused in ways that don’t map neatly onto ownership claims. Under a “Pay-By-Crawl” system, it’s entirely possible—likely, even—that one party could profit from content created by another, just because they host it or register it as theirs. This opens the door to mass appropriation of public knowledge under a monetized wrapper.
Worse still, this initiative has been cheered on by many who typically oppose centralized control online. Those who decry the “centralization” of DNS and content delivery networks are now embracing a system that would, in effect, centralize access to the public web through licensing gateways and paywalls. It was only yesterday that critics raised alarms about the gatekeeping power of infrastructure companies like Cloudflare—and now some are celebrating its new role as public web data tollbooth.
The future of the Internet, and the development of generative AI apps, depends on preserving open access to knowledge, not enclosing it behind licensing schemes that are designed to benefit large AI firms and a handful of intermediaries. Rather than building a better commons, Cloudflare’s proposal adds a layer of privatized governance and opaque monetization. Instead, it has chosen to profit from the very copyright logic it once resisted effectively helping to entrench the same extraction systems that generative AI critics are fighting against.
Cloudflare’s new “Pay-By-Crawl” model sets a dangerous precedent for how we control access to online knowledge, and risks reinforcing gatekeeping structures that threaten the open Internet. What’s especially disheartening is seeing long-time advocates of Internet openness applaud this move even as, until recently, they were raising concerns about Cloudflare’s central role in web infrastructure. That role, and whether it constitutes harmful centralization, remains heavily contested. But what’s not in doubt is that this new model moves us further away from the principles of transparency, openness, and permissionless innovation.
