Re: I think I have to walk away. I'm petrified.
stop. . just stop calling him, texting him and any other forms of communication. Stop putting yourself in a position to be rejected. This best thing for you to do right now is to be self-protective.
If he had strong feelings about you he would call you and follow-up on plans. It sounds like he is pulling away - he might be seeing someone else, or intersted in someone else.
You can have the conversation with him if you want, but he is probably going to reiterate what he has said all along - he doesnt want a relationship, just a "friends with benefits" thing - but it sounds like he doesnt want that right now. It is very easy to fall in love with something that is "almost", and to have hope even it is false hope. Listen to your head, you know how men act if they have strong feelings about a woman - they dont let them slip away.
sorry to be so blunt, but i have been there. . done that - and what i have found is that men pretty much tell you about their limitations up front - they may call you, confide in you, sleep with you - but if they tell you that they dont want a relationship. . they dont.