David Pierce / The Verge:

A look at robots.txt, a good will-based social contract governing the behavior of web crawlers, as experts call for more rigid tools for managing AI crawlers  —  For decades, robots.txt governed the behavior of web crawlers.  But as unscrupulous AI companies seek out more and more data …

David Pierce / The Verge:

A look at robots.txt, a good will-based social contract governing the behavior of web crawlers, as experts call for more rigid tools for managing AI crawlers  —  For decades, robots.txt governed the behavior of web crawlers.  But as unscrupulous AI companies seek out more and more data …

Source: TechMeme
Source Link: http://www.techmeme.com/240215/p9#a240215p9