It's not so bad once you get used to it. I'd rather drive the Jeep in crosswinds than my Triumph Tiger - talk about involuntary lane changes! Driving into the wind or with the wind isn't the problem, it's the gusty crosswinds. Just pay attention and hang on!
Driving through the Mojave desert last year on a trip to the eastern sierras with kayaks on the top and extremely strong evening winds. Not fun. Had to drive at 50mph, anything higher and I would have caused an accident. It is totally fine as long as you drive slowly.