I took benadryl all through my first pregnancy because I had allergies to everything, and my son turned out great. My doctor also told me to take diphenhydramine (same active ingredient as benadryl) this pregnancy for insomnia. I am 6 months, and u/s's looked great. I think it's a pretty safe drug. I wouldn't worry so much, the beginning part of pregnancy is challenging and stressful enough, without added worries about all the stuff you did before you knew you were pregnant.
Congratulations, and good luck for a healthy and uneventful pregnancy!