I am not sure if it is "normal", but just think about it...you have something around your face for sometime, and if you have sensitive skin, it could become irritated. Does it happen with every girl? Or just the same one...are you monogamous?
Try washing your face with soap and water immediately after so it doesn't "sit on your face" for a while! ahhaha...no pun intended. Good luck