I feel terrible for you, honestly I do.
It sounds like he's confused as to what he wants. He's also being selfish, from what I can gather. I completely understand why you would want to go over and cuddle with him. But I really don't think it's fair to you, at all. Why would he want to go out with you and have more of an attachment, when you are there for him any time he is lonely? You cuddle with him and have sex with him...he has the best of both worlds. It's completely selfish of him to ask of you. And it really sucks, because you love him, so of course you want to be close to him if the option arises.
Maybe he does have feelings for you, maybe not. Maybe he cares about you but just doesn't want to be in a relationship...I mean, he has all of the perks, so to speak, and doesn't have to deal with the negative aspect of a more serious relationship. He could like spending time with you, which is why he brings you food and sees you at lunch, but just doesn't want to be in a relationship. Maybe you should talk to him about it.
Of course you can't get him off of your mind...how can you fully move on from someone who you keep cuddling and having sex with? I'm not trying to be harsh at all, but it's going to be impossible for you to move on if you continue to see him like that. It's going to prolong the pain, and you deserve to be happy.
Perhaps you can tell him that you love him, and that's where you stand. But unless he wants something more serious, you can't just come over and be his late night cuddle/sex person. You are
worth a lot more than that. He isn't being fair to you, and you should say something. If he still says he doesn't want anything serious...then maybe you should just ask that he not call or, or not answer when he does.
Of course I know all of this is a lot easier said than done. I honestly wish you all the luck in the world with this.