Tired of seeing paywalls? Blame AI.

An arrangement once existed between search engines and websites. Websites would open up and make their content easy for search engines to find and index, which made the search engines more useful and popular by improving their search results. In return, search engines would send traffic to websites via those results, which helped the websites generate readerships and revenue.

To be sure, this arrangement wasn’t always symbiotic. Search engines occasionally made websites jump through unnecessary hoops, for example, or made sudden, seemingly arbitrary changes to their super-secret algorithms that could dramatically impact a site’s ranking. But when it did work, it worked to everyone’s benefit.

That arrangement effectively lies in tatters now, and it’s all thanks to AI.

Earlier this month, Cloudflare CEO Matthew Prince revealed some disturbing statistics. Back in 2015, Google crawled just two webpages for every visitor that it sent to a website. Today, however, Google’s crawl ratio now sits at 18 webpages for every visitor that it sends out. Meanwhile, OpenAI crawls 1,500 webpages for every visitor and Anthropic crawls a whopping 60,000(!) webpages for every visitor — and those numbers represent a dramatic increase from even just six months ago.

In other words, these companies are keeping more and more web traffic for themselves instead of sending it out to the sites that actually provide the original content they use to generate their results and summaries. This is terrible all-around, but it’s especially harmful for sites that rely on advertising since they only make money if people actually visit and see their ads.

As Nick Heer points out, this situation will likely force more publishers to protect their content with paywalls. Often used to encourage subscriptions, paywalls have become an increasingly common feature of the online landscape. But publishers may soon come to rely on paywalls for their very survival, by hiding content from AI companies that gobble up massive amounts of data to train their AI models sans any remuneration or compensation.