Yes, if you wait it might be too late. Also, even if it doesn't have to be set they will just splint it for you to give it support and help with healing. A doctor can also prescribe something for pain if you need it.
I broke two fingers a few years ago and could not move them at all. I was just going to wait an see what happened, but they very rapidly swelled up and turned very black and blue. I went to the hospital and wound up needing a hand surgeon to set and splint both fingers. You think that you broke it, but it's possible to have a dislocation or a displaced fracture like I did (I had both). Both of which need to be set in order to heal right. If you haven't gone to the emergency room by now, please do so. Or make an appointment with an orthopedist. If you call to make an appointment, tell them that you need an emergency appointment because you think you broke your thumb. They won't make you wait.