First of all, don't ever break up with someone due to what others say about him, in regards to looks and the like.
Only you know if you'd be happy with him.
Honestly, if your friends are willing to criticize and make fun of you over something as trival as your partner's looks....then I wouldn't call them friends.
ANywya, so in the end it should only be up to you. Does he treat you well? Are you both happy in the relationship? There are questions you need to ask yourself to find the answer.