The Visitor Center is 2 miles south of town. The road to Whitney Portal is at the stop light (only one in town) - it's about 9 miles to the Portal itself.

In Fishmonger's graphic, the new visitor center is the long building with the silver roof.