If you're not ready, you're not ready. As anxiety/panic sufferers know, it can strike from nowhere, which is really the hidden fear that makes us not want to go on vacation.
I just came back from a vacation last week. Before I went, I wasn't 100% sure I could do it, but I felt decent enough to try. A month before that, there was no way I could do it. And I knew it.
Therapy, meds, and time helped me in that time frame get to where I could handle it. Maybe a month from now, you'll be good to go.
I wish you weren't so hard on yourself. You are sick. If you just had a heart attack, would you think a vacation would be in order? You'll be better soon and all this will be a distant memory.
Good luck on your recovery.