I definitely don't think it's you. My hubby is having the same trouble with condoms. We've been together 15+ years, and I was always on the pill, so we never used condoms. Well, I went off the pill back in March, and we attempted using condoms. No matter what we try (and we have tried everything!!!), he loses his erection with the dang thing on.
I don't have advice on what to do (hubby's getting a vasectomy in a couple weeks so it's just oral sex for now), but I really don't think it's you.
Best of luck in overcoming the condom issue