Anxiety can definitely affect your sexual performance. It's a classic symptom. It tends to create a snowball effect: you're anxious about performing, so your performance suffers, and thus you become even more anxious, etc.
If you're on medication for your anxiety, that can also make reaching orgasm pretty difficult. SSRIs are notorious for this. Some men like the effect because it makes them last longer, but others don't care for it.
The only advice I can offer is to explain to your partner (maybe before sex) that you're the nervous type, etc. Also, I think going without sex for longer periods makes things work more smoothly when you do have it. (Given the current state of my sex life, that's not a problem for me.
Anyway, I hope this helps somewhat.