Skip to content
News Nieman Lab Nov 2026

Anthropic, OpenAI, Google, and Microsoft back a shared robots.txt opt-out for AI scraping

On November 2, 2026, Nieman Lab reported that Anthropic, OpenAI, Google, and Microsoft have each agreed to support a single robots.txt signal that publishers can use to opt out of AI training scrapes — covering all four companies at once rather than blocking each crawler separately.

The signal was proposed as an IETF (Internet Engineering Task Force) draft by Anthropic engineer Kevin Kelley. Anthropic already supports it. OpenAI says it will do so “in the coming months.” Google and Microsoft say they will follow next year.

Why this matters for publishers

The current arrangement requires publishers to block each AI company’s crawler by name, one at a time. As the number of AI companies and their crawlers grows, that approach becomes increasingly difficult to maintain. A shared signal reduces that burden: a publisher states one preference and, once all four companies honor the signal, it covers their crawlers automatically.

The announcement came during a period of public pressure on Meta, which had proposed commercial terms that legal experts said would allow broad scraping of content from business accounts. Meta agreed to delay implementation and pledged to consult publishers before finalizing the terms. Meta’s communications VP called the shared opt-out signal “an important step,” though Meta itself was not among the four companies that endorsed it on November 2.

Context

The robots.txt standard was not designed for AI training use. Existing directives block crawlers for search indexing but have been repurposed by publishers trying to manage AI access with the tools available. A formally specified opt-out signal for AI training scraping would be a clearer and more durable instrument — if the companies that have committed to it follow through on their stated timelines, and if the broader industry adopts it.

For publishers, this is a signal worth watching rather than a resolved question.