We should always be concerned about cancer, but I don't think that it would be necessary to worry in your case. Largely cancer is preventable. You can read about it on the internet. If you follow a sensible diet, exercise, don't smoke etc., statistically you're a lot better off.
If lipoma would be considered a fatty non-cancerous tumor, then I've had these lumps for more than 30 years.... Also there are so many different types of cancer. Some are mild and others are very very dangerous. I've also had skin cancer for more than 20 years. It was the mild variety. It seems like I had that purple spot "forever", before my doctor spotted it and cut it out.
So, I hope that's helps. Don't worry.
Just listen to your doctor and take a sensible approach to cancer. It is something we take seriously but without stressing out over it.