It is probably *because* of your trying. I never forced anyone onto #xmpp.
When someone asks I say “this is where I can be found” and that's it. In fact, it is the *not* inviting people to join you that gets them interested (this is a marketing psychology technique as well, I discovered later, but in my case it's just that I genuinely don't give a toss).