From a technical point of view, there is no neutral position when discussing federation.
This is a rare case of a genuinely binary choice: an instance either allows federation with neo-Nazis like Gab, or it doesn't.
If an instance allows federation with neo-Nazis, it is increasing the real life risk to vulnerable groups, because such federation makes it easier for neo-Nazis to recruit people to their cause.
It is reasonable to defederate with such instances.