I dont think you have much to worry about , I take altace but 20 mg per day .I do have high blood pressure .I have even accidentally taken an extra 10 mgs with no bad effects .it will not hurt you taken in that small a dose and it sounds like your doc just wants to keep it down. Blood pressure will vary throughout the day affected by many things . The best thing you can do is talk to your doctor about it. Im sure he will alleviate your fears.
hope this has helped