It's worth investigating, that's for sure. We had to move out of our last home because I kept having really bad allergies and asthma attacks. It turned out there was a ruptured pipe from the shower that was causing water to leak into our walls right into our bedroom. The mold was so bad that mushrooms had actually started to grow from the carpet.

(I didn't believe it either when I saw it). The landlord put a hole in the wall to fix the pipe and all that yucky stuff from inside the walls started to seep out. Over the next few weeks, I had to see the doctor a number of times for nausea, vomiting, chest pain, fatigue, and my asthma flared up so bad that I needed several breathing treatments. We couldn't take it anymore and decided to move, and as soon as we got settled into another apartment, my symptoms started clearing up.
I believe that mold can and does make us sick- it all depends on how sensitive your body is to it and how it responds. The only thing that seems strange to me is that you mentioned you've had the mold for years and you only recently started getting sick recurrently. Even so, I would see a doctor, to eliminate further damage that may happen to your body later in life.