There is no neutral position here. If you are a Fediverse instance owner, your choice is to either shrink or widen Gab's audience.
If you allow federation with Gab, you widen Gab's audience, you make it easier for them to recruit more neo-Nazis, and make it harder to contain them on a single instance.
The danger people are worried about is to vulnerable groups in the real world: actual physical danger.