I would talk to your general practice physician. They might have an idea of what's causing it to swell. If he thinks it's due to some sort of infection in the vaginal area, then you should see the gyno to get it checked out. It sounds to me like there is an infection somewhere. You could just be asymptomatic, which isn't rare. For example there are women that get bacterial vaginosis and never experience a symptom (in case you don't know, BV is a VERY common infection that occurs when the vaginal pH gets off balance and allows an overgrowth of "bad" bacteria). Anyway, you should get checked out to rule out any type of infection. If left untreated, infections like BV can lead to pelvic inflammatory disease, scarring, etc. Best wishes to you! Keep me posted on how it goes!