It might be more effective to penalize instances with open sign ups that the bots are using, to pressure such instances to use bot-filtering tactics on their sign up pages.

Rate limiting, blocking DMs, or defederating (perhaps graduated after a warning) from minor instances that can’t control bots sounds like a reasonable admin action.