I believe either Discover or Scientific American magazine recently had an article where this guy had breathed in some sort of spores while cleaning a vine off a house. The vine had all sorts of toxin on it, but he didn't realize it. He had asthma, and the spores he breathed in lead him to fatigue over the next few days. When he went in, the doctors found that he had spores growing inside his lungs, causing the wheezing and fatigue that he just suddenly felt. Apparently this is terribly uncommon, but happens when asthmatic lungs provide this perfect environment--like a petri dish--and the spores tend to grow really fast. The article stated that it could also happen in otherwise healthy lungs. They gave this guy some very serious antibiotics, and he got better quickly.
Now, this may not be happening with you.
But I would advise going back in to see your physician if this continues. It could just be an allergic reaction, or it could be something related to the allergic attack. (Or it could even be anemia or chronic fatigue syndrome, who knows?) Best to see your doctor if you're still feeling majorly fatigued.
Get better soon, Alyssa! Being a mom is the toughest job there is!