My disability is physical too (in my back) and I was also sent to an appointment with a psychiatrist. I even asked the psychiatrist why I was there and he told me that it was just part of the process.
It lasted about 40 minutes, and the psychiatrist was very nice. He asked the usual questions: who is president, what city are we in, did I hear voices and was my childhood happy.
I was very surprised and happy that he spent half the time asking about my daily living and how difficult it is for me to do simple daily tasks. I was not expecting him to ask about that, and it made me feel like he understood how difficult life is for me.
Go to the appointment - it probably won't be that bad and it will help your case.