Reddit restricts Internet Archive access over data scraping concerns

Reddit

Reddit has announced it will block the Internet Archive’s Wayback Machine from indexing the majority of its content following incidents where AI companies scraped Reddit data via the archive. Going forward, the Wayback Machine will no longer be able to crawl Reddit’s post detail pages, comments, or user profiles. It will be limited to indexing only the Reddit.com homepage, effectively restricting archived data to popular headlines and posts on any given day.

A Reddit spokesperson, Tim Rathschmidt, explained that while the Internet Archive serves the open web, some AI companies have violated platform policies by scraping data through the Wayback Machine. Reddit is therefore restricting access to protect user privacy and uphold its content policies, including the removal of deleted material.

These restrictions began to take effect recently, with Reddit notifying the Internet Archive in advance. Rathschmidt also noted ongoing concerns about scraping activity originating from the Internet Archive in the past.

Reddit has a history of limiting access to scraping tools amid increasing misuse by AI firms. The company offers data access to those willing to pay, having established agreements with Google for search and AI training data. In 2023, Reddit blocked major search engines from crawling its data unless paid and implemented controversial API changes that led to the shutdown of some third-party apps due to their abuse in AI model training.

While Reddit maintains a partnership with OpenAI, it filed a lawsuit against Anthropic in June, alleging continued scraping despite Anthropic’s claims to have stopped.

Mark Graham, director of the Wayback Machine, stated that the organisation maintains a longstanding relationship with Reddit and that discussions on the matter continue.