That's not on. Most women are sexy. Except the ones I'm related to, and any that look like The Donald. This douche is being passive aggressive enough that you'll feel you can't do any better than him. You can, and you will, but first you need to ditch this asshat so you're free to move on. When you're done, rally the troops, and go have some fun.

OP, here's hoping that he doesn't have a side chick yet stays with you for other, selfish reasons nonetheless. Either way it doesn't seem like a favorable situ for you if he's doing you the favor of being your BF yet fails to adequately communicate with you about important interpersonal relationship topics like sex. I hope that you're not settling because of low self-esteem issues. Good luck!