But if the issue is SEO bots then robots.txt would block the search engines thus meaning the spam content is of no importance (it's effectively private) and doesn't cause issues for the main sites SEO (nor help the spammers).
To be honest, I'm not certain of the purpose of the spam.
Some portion of it would end up in the search results, sure.
But I don't know if there's some secondary benefit to, say, a casino showing a link coming from my site even if my site has a robots.txt saying that the address for that link isn't to be directly indexed.
Is there such a benefit? If not then I'll just set up the robots.txt and observe whether that does indeed solve the problem. But I'd much prefer to just set up the permissions I know I want on my own running instance than spend time making inferences about the reasons bots are abusing my instance's inputs.
