Mike Bell is 6'4" and Ricky Carmicheal 5'4" so I think ultimately height doesn't make any difference. However, the MX guys are usually about 5'8" or so and wiry,think Bob Hannah, and the offroad guys a lot bigger.David Knight is 6'4" and a fast guy from my region, Corey Graffunder, is also about the same height.
I'm almost 6'5" and found that my height gave me more of an advantage in offroad events then MX.
My oldest son is 6'3" and in his first downhill mountainbike race he was 2nd overall and won the second event, he was as much as 8 seconds faster then the second place Pro on some parts of the course. Now if only I could get the runt back on a motorcycle.