This has NOTHING to do with you, your body, or your skills in the bedroom.
The only way to overcome this hurdle is to sit him down and have an open discussion.
Tell him how much you like him, and explain that you are just trying to understand what is going on. This about finding solutions; not harping on problems.
This wasn't a problem that developed over time: It's been THE problem from the very beginning.