Just found out that the simple Disallow:/ robots.txt I had set up initially was probably undone by upgrading Discourse.

Discourse doesn’t have an easy way to modify robots.txt, but instead I found a setting blacklisted_crawler_user_agents - which I now set to *. That should do the trick and remain resistant to upgrades.

Good to keep an eye at though.


this means that you don’t want crawlers to index this forum? why?


I think @darioha and I talked about this topic when setting up this forum, but I don’t remember the details.

I’m not against having meta indexed, but I feel people will feel more free to discuss things here if it’s not indexed. And some people may not want to contribute at all.

Happy to hear what y’all are thinking about this.


Just thought of another good reason not to have it indexed: spam bots won’t care about spamming here if it’s not indexed.