[tomek@vz]

What do SourceHut, GNOME’s GitLab, and KDE’s GitLab have in common, other than all three of them being forges? Well, it turns out all three of them have been dealing with immense amounts of traffic from “AI” scrapers, who are effectively performing DDoS attacks with such ferocity it’s bringing down the infrastructures of these major open source projects. Being open source, and thus publicly accessible, means these scrapers have unlimited access, unlike with proprietary projects.
These “AI” scrapers do not respect robots.txt, and have so many expensive endpoints it’s putting insane amounts of pressure on infrastructure. Of course, they use random user agents from an effectively infinite number of IP addresses. Blocking is a game of whack-a-mole you can’t win, and so the GNOME project is using a rather nuclear option called Anubis now, which aims to block “AI” scrapers with a heavy-handed approach that sometimes blocks real, genuine users as well.
The numbers are insane, as Niccolò Venerandi at Libre News details.