I don't think there's anything wrong with it. My first watched tv all the time when she was a baby. Like your baby, she just seemed to like the flashing colors and such. I cut her back to one or two shows (half hour long) a day when she was a toddler, and then I had another baby when she was two, and she got to watch a little more tv each day while I was getting used to having two kiddos. Then I cut her back to just one or two shows a day again. She's 5 1/2 now, and her sisters are 3 1/2 years old, and 12 months old, and they only watch tv occasionally.... usually when I want them to have a rest, if they're sick, or if I need some quiet time in the house. They love tv.... and they'd watch it all day if I let them, but I don't let them and they don't argue about it.
I guess my point is that taking the tv away will only be as difficult as you allow it to be. I wouldn't stress about it right now. At this point, whatever works for cutting out your stress load is what's best for you and your baby!