I think that there are many things that help add up to the fact that you will/are having a big baby. Me nor my husband has big baby's in our family, but of course me the smallest in my family (I'm almost 5'4"), and everyone else is over 5'9', had my first baby which was 9lbs, and only +2 days over due. I was always on track with my fundal measurements with my first, but I can say that I always had sugar in my urine, but no diabetes. I am now 28weeks, and measuring +6 weeks ahead, still sugar in my urine, no diabetes. I go for an ultrasound soon to get an estimate of the growth.
I think that my doctor, like others, take everything into account to come to a conclusion that you are having a big baby, like my situation, 1st baby large, fundal measurement large, sugar in urine....at lastley, if the ultrasound/growth is measuring big.
Then they can say that yes, you are having a large baby.