I also have tried it both ways.
I live in Sacramento. Mapquest says "6 hrs 28 mins / 335.20 miles".
The two times I drove thru to home were the two most miserable drives in my LIFE! I barely remember them and looking back I don't see how I made it. I certainly don't remember much of those drives!
** serious question ** What's more dangerous, the mountain activity, hiking, climbing etc, or the drive to and from the mountain?
Another two trips back from Whitney I stopped at the side of the road at the 120/395 junction and slept the night. Completely transformed the drive back into a glorious experience, sleeping in the car with the moon lighting up the Mono Lake Basin, waking up as the sun starts to warm up the car, breakfast at Whoa Nelli, morning light (something I don't normally see) on Lee Vining Canyon, Tioga Pass, and the rest of Yosemite, arriving home in the middle of the day, all makes the end of a trip a completely different experience.