I have ridden DOWN both Whitney Portal and Horseshoe Meadows roads. They're a kick! My daughter rode down WP Rd with me 3 years ago (she was 11 at the time) -- we kept the speed down because she was less experienced, and I worried about over-heating caliper brakes. On my own, I'd let the wind modulate the speed.
Riding up... great workout, but I haven't done that. I have occasionally seen riders going up. Traffic is so light it shouldn't be a problem.
There are miles of dirt roads on the north side of Whitney Portal Road -- Movie Road-Hogback Road ( Map )
on BLM land, but they would only be fun early mornings when it's cool.
Go for it!!