He is joking - agent orange is/was a controversial chemical used during the war in Viet Nam that allegedly causes some nasty long term effects.
While rare, some women are allergic to their mate's semen, but the man being allergic to his own semen is new to me. Were all the condoms you tried latex? If so, try the one made of a plastic material, and you might even try the so-called female condom.
He should probably have a semen analysis done as well.