Opens in a new tab

How to keep the web open in the age of AI: Crawling, AI and the traffic angst

A few weeks ago, Lisa Macpherson of Public Knowledge and Anna Tumadóttir of Creative Commons published an article called “Creating a New Internet Bargain to Preserve Fair Use and Protect the Commons” in Tech Policy Press. I focus on it in this piece because it gathers nearly every argument I have heard on this topic over the past three years. Also this week at the IETF, the AIPREF and Web Bot Auth groups take up many of the issues it raises. If you are interested in IETF AI pref where we discuss how site operators can signal to crawlers about the use of their content for AI and also web-bot auth which discusses how to authenticate (and unfortunately one stream wants to authenticate through identification) then you are in for a treat cause I have been following those groups as well as doing my usual infrastructure and digital rights work.

Anyway, the other reasons to write this blog is the amicus briefs in NYT v. OpenAI, which focused on training and fair use and despite my awareness raising about the privacy and access issue, hardly anyone criticized NYT for seeking discovery of 20 million ChatGPT logs full of users’ private chat. The other reason for this blog is the pile of copyright comments to the European Commission. But since it was getting too long I thought I divide it to parts. So here is part one.

The argument in the Tech Policy piece (and so many others in this space) is that the old bargain of the open web, through which creators and publishers supplied content and platforms returned attribution and traffic, has dissolved. Generative AI takes away the eyeballs and therefore it is producing a new and unsustainable system that may benefit large AI developers and big rights holders and everyone else will be at a loss.

Let me be clear that I have shared a lot with these two organizations, and many organizations that defend the open web. I agree with anyone that defends AI training as generally fair use, opposes a long list of bills that could lead to its erosion, and says this is a market structure problem rather than a copyright problem. I agree with all of that. I have also worked on market concentration, especially in infrastructure and DNS, and I share their worry that the AI economy could consolidate power in a handful of companies and create new gatekeepers.

But (weren’t you waiting for that but?) I am also against solutions that re-intermediates the Internet and prevents a decentralized AI from happening. The piece makes a mistake that has run through this debate since it started. It treats the economic interest of publishers and creators as if it were the global public interest in access to knowledge. And several remedies its framing leaves on the table (but doesn’t fully support) such as bot identification, collective licensing and preference signals backed by enforceable rights, would produce exactly the enclosure it warns against. And it creates intermediaries around it (if you have to identify yourself, you might not independently go on a day of crawling, you use the data crawler if you can. if you are from a vulnerable community that doesn’t have the means… oh well, no crawling for you I guess). To be fair, the authors are reluctant to endorse any of them. But Creative Commons, in its comments to the European Commission, endorses bot identification, says it should be privacy-preserving, and then backpedals those privacy safeguards by saying they must not become a back door and takes blacklists at face value without questioning where these blacklists come from. (we prefer to call them blocklist) It says to develop bot identification practices with privacy safeguards in place that should not be abused (the privacy safeguards they mean) by black-listed deceptive bots.

In my opinion, the web is not being enclosed by crawling and scraping. Hess and Ostrom’s work on knowledge as a commons draws a distinction that this debate keeps overlooking. Knowledge is nonrival and my reading a page does not use it up and yes if my crawler uses up so much of your infrastructure that it crashes, it’s a real issue. The threat to a knowledge commons is not much about overuse than about enclosure, through licensing, intellectual property restrictions, overpatenting, overpricing, and the failure to preserve and archive (remember news corporations taking archivists to court?). What is rivalrous is the infrastructure that serves knowledge that can go down due to large traffic. AI crawlers can overwhelm servers, and that is a real problem we need to solve. But it calls for best practices about network capacity and resilience, not rules about who may read and crawl.

So maybe replacing the bargain with a licensing deal between big publishers and Big AI, will not preserve the open web. Identifying bots to enforce copyright won’t either; it may even lead to further enclosure. Nor will legally enforcing robots.txt and calling anyone who ignores it “a bad actor.” I shall resist the temptation to rant about unsophisticated good-actor, bad-actor arguments.

And if I don’t say this, you will all think my oracle wrote this on its own: in this whole conversation, the person the open web was built for, the reader and explorer keeps going missing.

The bargain was not great

The authors are careful not to call the exchange of content for traffic the “open web bargain.” But the piece assumes this exchange is what kept the web open. This has become the dominant narrative: tech companies got to index websites, and in return sent the publishers traffic.

I think the word “traffic” that we keep using in these conversations obscures what we are really saying. If we put the end user, our explorer, at the center, when publishers say traffic is falling, they mean they can no longer drag the user to their website to see the ads and the donation button.

But why should we have to drag users, sometimes even without their consent, to websites that are sometimes terribly designed? Why shouldn’t we work on quality of content so that the user goes to those websites on their own? The early, open web was built on open protocols and interoperability. It was built on the ability to publish and to access information without having to negotiate with commercial intermediaries and the governments, what some call permissionless innovation.

The bargain these debates describe came later, when large publishers became dependent on search engines and social media for discovery. Their revenue was mostly advertising and donation (this is to be contested as well) and it was tied to traffic which meant getting the user onto their pages.

This arrangement in my opinion, was never universally good. It led to many privacy and access issues. It rewarded publishers who could win at search ranking and ad tech (so basically those who had access to money and large editorial teams). It surveilled users, and on low-quality sites the ads led them to phishing and malware. It pushed everyone else toward SEO bait, engagement chasing and pages so ad-heavy that readers tried to avoid them. Hopping between websites and loading heavy pages was not accessibility-friendly either, which is partly why larger platforms built their own apps. Also referral traffic may matter enormously to some publishers, often the ones who are most vocal in policy circles. Not all publishers care about referral traffic.

To be clear, I don’t think the decline of this traffic means an open web principle has been broken. It may mean someone’s business model is no longer viable.

It matters to acknowledge this in our debates because the remedy we are building is based on how we frame this problem. If what broke is a principle of openness, we should restore openness. If what broke is a business model, we should be honest that we are being asked to rescue a business model. And you cannot restore openness with more copyright or more restrictions on access. And I am not into coming up with solutions to rescue business models that are becoming outdated. We should come up with solutions that keeps the Internet open and empowers the readers and explorers and open source developers to build and innovate.

Control-based remedies can create the enclosure

The piece names two forces behind the enclosure of the commons and this is also mentioned in other articles by Open Future and others. Extractive enclosure is what dominant AI firms do and it means lock up data through exclusive licensing deals and suing rivals over scraping. In extractive enclosure, some crawlers also override access signals and mask their crawlers (these two acts are not categorically wrong sometimes there are legitimate reasons not to follow signal or hide identity). Defensive enclosure is what rights holders and site operators do in response such as licenses, lawsuits, blocking, paywalls and calls for new legislation.

Here I agree that restrictions imposed by operators themselves can shrink the commons. What I disagree with is what actually does the shrinking, and I care a lot about the cause because the remedies that are put forward is invited by this sort of framing. The piece itself proposes few solutions, but others in this debate push identification, collective licensing and enforceable signals. I don’t think these solutions would reverse defensive enclosure. They would formalize it and as the authors themselves note, that does not lead to disintermediation either. And those who lose are not big tech and big AI and big media. The researcher’s crawler, the user agent fetching a link someone asked for, the small open source project and many good commercial initiatives will be left out and won’t have access. Carve-outs won’t work either. And I will explain why in another piece.

Maybe a tangential issue but the piece categorizes data poisoning under defensive enclosure, citing artists who use Nightshade to “ruin or poison AI models.” I don’t think that is defense. Defense has to be proportionate. Blocking a crawler is defense in some context but poisoning a model is offense, and its damage spreads well beyond its target. A poisoned file cannot tell an archive or an accessibility tool from a commercial scraper.

ABOUT THE AUTHOR
Farzaneh Badii
Digital Medusa is a boutique advisory providing digital governance research and advocacy services. It is the brainchild of Farzaneh Badi[e]i.Digital Medusa’s mission is to provide objective and alternative digital governance narratives.
Read more