Some guys are made this way and sometimes this can happen as a result of a hernia. It is important to verify that you do not have a hernia. If you have not been checked for one, it would be good for you to mention this issue to a doctor the next time you see one. It is probably not an issue since you do not have any problems like pain. However, if something changes, you should mention it to a doctor sooner than later just to be safe.
Good luck to you.