TechRadar · November 9, 2026 · On the brief until November 23, 2026

Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

Intellectual Property LawComputer ScienceMedia Studies

THE AI ANGLE

Scraping web data to train generative models

Google, Microsoft, OpenAI, and Anthropic agreed to honor a shared robots.txt rule that lets websites opt out of AI training. Webmasters can now set one preference instead of configuring rules for each separate company crawler. This system regulates content usage while still allowing bots to access and crawl pages.

Summary written by AI Business Lens with an AI model from the article at techradar.com. It is not the article, and the publisher has not reviewed it. For publishers.

THE TEACHING ANGLE

A key debate centers on whether voluntary protocols provide meaningful legal protection for creators when crawler compliance remains optional and access is not blocked.

Read the original at techradar.com Generate teaching or study materials

Instructors get discussion guides, assignments, and mini-cases. Students and readers get a plain summary, points to raise, an exercise, and a self-check. All built from the full article. A free account saves stories and gives you three generations. Stories stay on the brief for 14 days. After that this page keeps the link to the original.

Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

More in Intellectual Property Law