I agree that the switchbacks aren't too bad in isolation but by the time you reach them you have already done a fairly tiring hike. Also for people like me that Whitney is a huge adventure by the time you get to the switchbacks your excitement and adrenaline have waned considerably. They are also fairly tedious with no change in scenery which adds to the difficulty.