Actually I don't think it's unsual at all.
I've had panic/anxiety problems on and off for as long as I can remember....off right now, thank God
Every single time I ever got anxious or was about to have a panic attack the first thing I noticed was that I got very hot.
I felt like my temp had to be 105 degrees, when actually it wasn't elevated at all.
I talked to my Doctor about it once and he assured me that many people with panic or anxiety problems say the first thing they notice is that they either got extremely hot, or they got chills.
I was one like you, who felt like my temp was rising, while I have a friend who still suffers from panic/anxiety problems and she says the first thing she feels are chills.
This is indeed a strange disorder.
But, like with anything if it concerns you, you should talk with your Doctor about it.