Hi, It sounds like you are just not a naturally tactile person. I also would say that this is probably due to your upbringing, where emotions, feelings etc were suppressed.
Were your parents older parents, I say this, as my parents are in their 70s and it is a generation thing I think, my dad has never hugged me or told me he loved me, but I know that he does.
Remember on a positive thought, actions always speak louder than words.
If you find it so difficult, have you thought about writing him a note, explaining how you feel, and explaining how you just find it difficult to express yourself, due to learning about supressing feelings from your parents.
On the other hand you may also have a genuine fear of rejection, and scared to approach him, for making a fool of yourself.
I would start by writing him a letter, see how he responds to that, and once you feel more confident and natural around him, things will probably change.
Best of luck