A restrictive diet can be harmful if taken to an extreme, as in anorexia. I think most anorexics are vegans but not all vegans are anorexic. So vegans tend to get a bad rap by doctors who have had some contact with anorexics.
They might think, "here we go again, I have another anorexic on my hands".
I think the average American diet restricts healthy vegetables. If we're going to be restrictive, let's restrict processed foods!