I like leaving early and taking my time, after all it's a short day. This gives you the ability to stop at Lone Pine Lake, Bighorn Park and/or Trailside Meadow for extended period and still get to Trail Camp in time to set up your campsite and explore.
The other advantage of an early start is you avoid much of the heat.
I've never found travel up the hill to be an issue, there are usually more than enough folks going up that someone is usually in your line of sight. The problem is down when you are tired and not concentrating. It very easy to walk off the trail between Trailside Meadow and Mirror Lake, I've done most of my trips down to some degree and have seen many wandering around in search of the trail.
