Like people said, it depends on how old your bro is...if he's very young (7 or 8), then I wouldn't suggest hitting back. If he's a bit older (at least 10), then I say "If he bites ya, bite him back!!"
Using a little ad-libbing for the "bite" part of course!
If that doesn't work, and your mom won't listen, try this: if he tries to hit you or does land a smack, take your hand and put it on his head and squeeze slightly (as I said, depending on his age and tolerance for pain
) and then squeeze your fingers up into a wedge shape as you let go, puckering his scalp up. That, or grab him on the back of the neck and give him a small squeeze.
I know it sounds strange and cruel, but as long as you don't crush or suffocate the kid, then try these. Trust me, my cousin used those on me when I was in my hitting stage, and now once in a while I use the same tactics on my boyfriend if he gets on my nerves