Tolls are a way to pay for content in the AI era on the web

AI web crawlers had begun inundating news and information websites with thousands of requests a day compared to the handful they typically saw from search engines. Not only was the explosion in traffic ballooning hosting costs for these sites, the bots supplied zero traffic to them in return. Web traffic in exchange for permission to crawl has been one of the unseen foundations of the internet economy for a generation.

This sounds like a good idea. The AI companies pay if they want to access your site. It needs to happen at scale, a new way for people to get paid for what they write/make.

Today, Cloudflare Is Blocking AI Crawlers by Default

Last year, internet infrastructure firm Cloudflare launched tools enabling its customers to block AI scrapers. Today the company has taken its fight against permissionless scraping several steps further. It has switched to blocking AI crawlers by default for its customers and is moving forward with a Pay Per Crawl program that lets customers charge AI companies to scrape their websites

Cloudflare has scale. This might be the start of the toll era on the web.

← Back to Micro