Freedom of speech always comes with freedom of association. My own personal problem with instance blocks isn't so much the block itself, but that people are doing them automatically, without vetting, and usually without even consulting their communities, so the people don't know what they're not seeing.  That said I have always advocated a minimalist position when it comes to administration because it is the most practical position: the less you police, the less work you have to spend on policing it and the less social tension you create by doing so.  People are generally adults here, they can work out their differences themselves without intervention for the most part.