I also had a hard time when I started the Wellbutrin, It was a rough few days, only about 2 or 3. Then I was all right. But if the doctors are telling you it's not good for you and you should not take it, listen to them, we are all different. I did not get a panick attack, I just got depressed as I have not been in a long time.
I am O.K. now, but now that I think of it, I have had several anxiety attacks but not to often, they are usually caused by stressfull situations.
I don't think it's us, I think anyone can have them.
I hope your doctor finds a medication that will work better for you.