Why is the deal behind the move? Is he moving with family or what?
If he really didn't show any sadness or anything at all, in regards to moving away from you, then I would lean towards the answer that you're not a priority. Then again, this is just a guess, I don't know him or you, and nothing about the move.
You said he didn't want you to visit before school. I think that right there is a big sign. What reasons did he give you for that, if any?