AI firms promise a unified tool for publishers to block data scraping

OpenAI, Google, Microsoft, and Anthropic announced plans to provide a single, standardized instruction for media outlets and websites to block content scraping. The proposed solution aims to replace the current requirement where publishers must execute multiple blocking commands for each individual crawler.

The implementation timelines differ across the companies. Anthropic stated it has already started rolling out the measure, while OpenAI plans to introduce it in the coming months. Microsoft and Google intend to implement the protocol in 2027.

This initiative arrives amid mounting criticism and legal challenges from news publishers regarding bots collecting articles for AI models without permission. For instance, the New York Times filed a lawsuit against OpenAI and Microsoft in December 2023, alleging unauthorized use of millions of articles.

Tech representatives noted that opting out could reduce the likelihood of a publisher's content appearing in AI applications. Additionally, Microsoft clarified that scraping will remain permitted by default until the setting is manually adjusted by a publisher, and the new control applies strictly to future model training.