Think of the Inviter
I recently read the blog post Always invite Anna when it showed up on Hacker News.
As someone who is more introverted, I have been lucky enough to be on the receiving end of invitations throughout my life. But people should also consider the flipside.
For those in Anna's position, I would also say to think of the inviter (as long as you don't dislike them). They may be extending invites because they genuinely value your company. It could be that they're looking to build relationships in a new place. They could be having a rough time. Whatever the reason, remember that they're not always reaching out from a more secure position. Saying yes once in awhile might be helping them out, too.