Hey Mike--yes, in the "old days," the YMCAs were men-only, and swimming nude was the custom. I learned to swim that way--with my Dad, no less. No one thought anything of it. But when the Y became coed (late 70s-early 80s?) nationwide, the practice obviously stopped.
There very well may be some that maintain men's times for the pool, but I wouldn't count on it now.
The other guys are right--find a stream/river, a secluded area of a pond or lake--that's the best bet. Just be sure you are well away from anyone who might be offended, especially women and children.
There is absolutely nothing "immoral" about swimming naked, provided you do it either in an area where it is customary or where no one would be offended. Personally, I hate wearing swim trunks and I swim without them whenever possible. But I also would never remove them unless it were acceptable.