You need to relax....breathe in breathe out.....
The best way to push a man away is to act clingy or be constantly worried about the relationship.
Sometimes in life we have to let go and let nature take it's course. Stop for a moment and smell the roses.
I know it is easier said then done and I myself am guilty of easily getting worked up, but with hard work I have found a place of comfort and peace and thats what you need to find.
Let this relationship happen as it happens. Be true to yourself, treat this man how you would want to be treated and if it is meant to be it will fall into place. Worrying about the future with this guy will not have any change on the final outcome. Basically worrying is a waste of time and energy and only has negative consequences.
If you can't get to a point of calmness and sit back and enjoy the present moment of the relationship then maybe you might want to consider meds to help decrease you anxious tendencies a little.
Good luck and I hope all works out.