Conversation

Dark Visitors maintains a list of AI web agents (chatbots, scrapers, search crawlers) and provides an API for updating your robots.txt

https://darkvisitors.com

@jonny and I said "mhmm", so we made a github action that regularly creates PRs for our mastodon fork if there's any updates

https://github.com/NeuromatchAcademy/mastodon/pull/41/files

and here's a version with comments to help you make it work for other websites

https://gist.github.com/hobgoblina/c885b7f502307802f2b034fabf312223

consent shouldn't be opt-out, but ai bros are dicks. so here we are

#ai #nobot #mastoadmin #mastodon