My doctor told me too that ear infections are usually accompanied by fever, but I have heard people say that their lo's never got a fever despite many infections so who knows..
If your lo is fussier than normal and tugging at his ears I would personally take him in to be checked. It's hard to know though, because both these things can also be teething. If he's just tugging at his ears though, I would wait a little bit before running him to the doc, and see if it subsides on it's own.
Whiskers