Unless he urinated before going back in with the tip of his penis, there was probably semen in his urethra which could have gotten into your vagina. Washing the outside of the penis really doesn't take care of the more important semen still in his penis so that isn't much added protection.
It's probably too late to take the morning after pill (but check, just in case) so you'll just have to wait and see. It's not as high-risk as if you'd had full intercourse, but there definitely is risk. Next time, make him keep it out!