Well, this is really something that you and your husband need to communicate about. I know that he is being really evasive and casual, but you do need to get to the bottom of this.
Maybe there are some men who really do just stop wanting sex. Maybe he has some kind of problem with getting turned or getting an erection but it's not bothering him. Maybe he is gay and just using you as a beard. Or maybe he really is having an affair (gay or straight). Or maybe (and I do not mean this to be at all insulting) he just doesn't get turned on by you anymore. It's not a great thing to say, but for me, having been with my boyfriend for only two years, the idea of sex with him doesn't really do much for me anymore.
So, you really need to get an answer here. If he is cheating, or if he is gay, then you need to find out so that you can move on with your life. If it something like he really just doesn't want sex or else has some kind of impotency problem then you need to work through that as a couple and come to some kind of conclusion. And if it is that he doesn't get excited by you anymore, you need to either find a way to make your sex life thrilling again for the both of you or else you should move on to someone who DOES want to be physically intimate with you.
There could be other things, too. (Really, there could be a hundred different things). Some men are addicted to porn and that does it for them, they don't need or even desire a real life partner. Or maybe he is on some kind of drugs that dull sexual desire. Now, before you freak and think I'm calling your husband some kind of porn-obsessed drug addict, I am just throwing every possibility I can think of out on the table here. But the truth is that only your husband knows, and you deserve to find out. Good luck.