First thing. Don't let anyone (even your mom) guilt you into anything. I suspect you were "guilted" into the agreement in your 20's, anyway.
Having been a caregiver for my wife for a year and a half before she died, I can tell you that it is the most stressful thing I"ve ever been thru. And I knew the end was not far away. I cannot imagine looking forward that the next ???? years if my life would be like that.
It will destroy your marriage, your work life, forget about a social life. Caregiving WILL BECOME your life. And you are way too young for that to happen. You will feel like you are in a prison and resent every minute of it.
There are other care possibilities out there. Explore them and find one that will help you fulfill your responsibilities to your brother, but that will be fair to you and your family.
K