I spend a lot of my working life inside other people’s websites, and lately inside my own. So a small piece of news from the end of last week caught my eye, because it is about who gets to decide what happens to the words and pictures on a site after a crawler comes to visit.<\/p>\n
On October 30, Google, Microsoft, OpenAI, and Anthropic said they will back one shared instruction in a site’s robots.txt file for opting out of AI training. The token is called Until now, I have had to do this one company at a time \u2014 one entry for GPTBot, another for Google-Extended, another for ClaudeBot, and so on \u2014 and then keep that list current as crawlers are renamed, split, or replaced. One line that names a purpose instead of a bot is the whole idea.<\/p>\n First, it lowers the bar to almost nothing. A small site owner with no developer and no legal team can copy one block into a text file and be done. The old list approach worked fine for big publishers; for the rest of us it was homework that never stayed finished.<\/p>\n Second, a purpose-based rule survives change. Bots get renamed. Companies launch new crawlers. A rule that says “not for training” does not care what the crawler is called this year, and that is simply better design than chasing names.<\/p>\n Third, the companies agreeing are the ones that matter most to most publishers, and they agreed on the same token instead of shipping four competing dialects. One syntax, in the one file site owners already know, is a real deliverable \u2014 and a clear, dated refusal on record is also useful evidence if a publisher ever ends up in a licensing argument.<\/p>\n Now the other side, because there is one. To start with, it is not in force yet, and the timelines do not line up. Anthropic says it has begun implementing it. OpenAI talks about the coming months. Microsoft and Google point to 2027. Meta has expressed support with no commitment at all. An announcement is not an enforcement date, and in the gap a publisher who only adds the new line and deletes the old per-bot rules has actually weakened their position \u2014 so the old rules need to stay until each company confirms it is honoring the new one.<\/p>\n It is also still just robots.txt, which has always been advisory. Nothing technically stops a crawler from ignoring it. It works because the big players have reputations to protect, and it does nothing at all about smaller scrapers with no reputation to lose, or about copies of a site already sitting in somebody else’s dataset.<\/p>\n And it does not undo anything. It governs future crawls. It does not remove a site from models already trained, and it says nothing about paying for material already used. There is also a quiet split hiding in the details: Google plans to read the shared token as covering grounding \u2014 a model pulling live pages to build an answer \u2014 as well as training, while Microsoft treats grounding separately. So the same one line can mean slightly different things depending on who is reading it, which is exactly the kind of confusion a shared standard was supposed to end. Finally, note that a wildcard block does not trigger it: I have to name the token explicitly.<\/p>\n I am for it, with my eyes open. I will add the line to the sites I run, keep every existing per-bot rule in place beside it, and write down the date I added it. That costs me five minutes and states, in the clearest common language available, what I do and do not consent to.<\/p>\n But I am not going to pretend a text file is a fence. It is a sign on the fence \u2014 one the biggest companies in AI have now publicly agreed to read the same way. That agreement is worth having, it was overdue, and it makes the next conversation, about what happens when the sign is ignored, a much cleaner one to have.<\/p>\n","protected":false},"excerpt":{"rendered":" I spend a lot of my working life inside other people’s websites, and lately inside my own. So a small piece of news from the end of last week caught my eye, because it is about who gets to decide what happens to the words and pictures on a site after a crawler comes to […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-165","post","type-post","status-publish","format-standard","hentry","category-technical-commentary"],"_links":{"self":[{"href":"https:\/\/quentinpidcock.com\/content\/wp-json\/wp\/v2\/posts\/165","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/quentinpidcock.com\/content\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/quentinpidcock.com\/content\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/quentinpidcock.com\/content\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/quentinpidcock.com\/content\/wp-json\/wp\/v2\/comments?post=165"}],"version-history":[{"count":0,"href":"https:\/\/quentinpidcock.com\/content\/wp-json\/wp\/v2\/posts\/165\/revisions"}],"wp:attachment":[{"href":"https:\/\/quentinpidcock.com\/content\/wp-json\/wp\/v2\/media?parent=165"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/quentinpidcock.com\/content\/wp-json\/wp\/v2\/categories?post=165"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/quentinpidcock.com\/content\/wp-json\/wp\/v2\/tags?post=165"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}__usage_ai-training<\/code>. I put that token in a robots.txt group with a Disallow rule, and every participating company is supposed to read it the same way: do not use this site to train your generative models. The proposal was written up as an Internet Draft by Kevin Kelley of Anthropic and submitted to the AI Preferences working group at the IETF, the body that handles internet standards. Reporting on the announcements is collected in this rundown of the new standard<\/a>, which also notes the important limits below.<\/p>\nThe Case For It<\/h2>\n
The Case Against It<\/h2>\n
Where I Land<\/h2>\n