I spoke to my doctor about this because I was nervous because I had about two cold cut sandwiches and two hot dogs after I was pregnant. When I told him that I was worried he just laughed. He said that listeria is a concern, but it shouldn't dictate everything we eat. He said if we lived in Mexico or somewhere like that, then he might be more worried, but he said basically there was nothing to worry about. His wife just had a baby 6 months ago, and he told me that all he advised her to do in regards to her diet is avoid fish, and eat things from all over the food pyramid.
So don't stress about what you have already eaten, seriously no harm done. I avoid these foods because I am a worry wort, but I don't stress about it. If we were to avoid every type of food that could have listeria in it, we would not be eating anything, including fresh fruits and veggies.
Hope that makes you feel better!