Your notes are more correct than your wrist watch. I've recorded the tracks before with an accurate GPS. I can't seem to find the file right now, but I believe the mileage was just a tad over 5 miles even one-way. We didn't veer off course though. Even though you guys went off a little, there's no way it added up to 4 miles.
This is actually less than the 4 miles I've heard from other folks. I've been within a eighth of a mile of "official" numbers on other trails, and I was within 5 ft of the 14,505 quoted height for Whitney so I'm inclined to trust my Garmin's readings.