According to Dark Visitors founder Gavin King, most of the big AI agents still follow robots.txt. “That was consistent,” he says. But not all website owners have the time or knowledge to regularly update their robots.txt files. And even if they do, some bots will bypass the file commands: “They’re trying to hide the traffic.”
Prince says Cloudflare’s bot ban won’t be an order this type of bad actor ignores. He says: “Robots.txt is like putting a sign that says ‘you are innocent.’ “This is like having a real wall guarded by armed guards.” Just as it flags other types of suspicious behavior on the web, such as price-scraping bots used to monitor illegal prices, the company has created processes to detect even the most carefully hidden AI crawlers.
Cloudflare is also announcing an upcoming marketplace for customers to negotiate terms of use with AI companies, whether it includes payment for using content or trading credits to use AI services to trade. “We don’t really care what’s done, but we think there should be some way to give value back to the original content creators,” Prince said. “Compensation does not have to be in dollars. Compensation can be credit or recognition. It could be a lot of different things.”
There’s no set date for launching that marketplace, but even if it launches this year it will be joining a crowded field of projects aimed at simplifying licensing agreements and other permission systems between AI companies, publishers, platforms, and other websites.
What are AI companies doing about this? “We’ve talked to a lot of them, and their reactions have varied from ‘this is reasonable and we’re open’ to ‘hell,'” Prince said. (However, he would not say the names.)
The project changed quickly. Prince cites a conversation with Atlantic CEO (and WIRED editor-in-chief) Nick Thompson as inspiration for the project; Thompson discussed how many different publishers have encountered secret web scrapers. “I love that you do it,” Thompson said. If even big-name media organizations are struggling to cope with the influx of scrapers, Prince reasoned, independent bloggers and website owners may have an even greater difficulty.
Cloudflare has been a leading web security company for years, and provides a large part of the infrastructure that hosts the web. It has historically remained as neutral as possible regarding the content of its websites and services; on the rare occasions that it has made an exception to that rule, Prince has insisted that he doesn’t want Cloudflare to be the arbiter of what’s allowed on the Internet.
Here, you see Cloudflare in a special pose. “The path we are on is not sustainable,” said Prince. “We hope that we can be a part of ensuring that people are paid for their work.”