I feel your pain. The year I had to turn my wife and daughter around at about 14K', we had originally planned to leave Consultation Lake at 5:00 am. My daughter, 11 at the time, was knackered from the backpack up the day before and we let her sleep in. Didn't hit the trail till 6:30 the next morning and that made all the difference in the world. If we had left at 5:00 as planned the icing issues above the chute wouldn't have been a factor. Everyone was well-acclimated and feeling strong, but the footing was getting treacherous.
I deeply regret starting late, but never the decision to do a one-eighty when safety became an issue for my girls. When I told them we had to turn back my wife completely understood, but my daughter just dropped down and started sobbing. That was tough - she was SO determined to summit. They tried again in 2013 but were monsooned off the mountain at Trail Crest.
Live and learn. From that point on start time was never again negotiable.