You have nothing to worry about. Most guys are your size. Anything from 5.0 inches to 6.0 inches is the norm, or the so-called "average," which ~85%-90% of all men (of all races) fall into.
I am currently in college. I used to play varsity baseball (3 years) in high school, and, for a single year in college. Trust me, they won't say anything, because, well, many large looking guys (6'0 feet tall and over) are not exactly well-endowed. Everyone, except for a select few, are average -- the well-endowed individuals tend to be more of an exhibitionists.
For the first week, some guys will look at your package out of curiosity (to compare), but after that, everyone tends to their business. If not, I'd question his sexual orientation.