Maybe a blocked duct?
That doesn't really make sense though since it's not sore while on your period, I would think it would be sore all the time if it were that, mine were.
This one is baffling.
I think if it were me I'd go ahead with any tests your doc wants to do, just in case. It's possible there's something there, whether it be cystitis or what have you, & it's just not large enough to feel manually yet.
I have a cousin who had breast cancer & had no pain & had gone to her doc because there was a rash on her breast that wouldn't go away. No pain, no lumps.
It's very strange the way these things work. Some people have lumps, but no cancer.
I think I'd have it checked to be on the safe side.