In this picture, the other party memeber (not me) has not accepted an invite, but his icon does not correctly indicate this.

I believe if someone is invited, their icon is supposed to be a loading icon to help players understand if the party is ready to queue or not.