I would definitely agree that paddling an outrigger well in contrary conditions requires balance, although you can paddle one decently without too much in the way of balance skills. Flying the ama requires some rather exceptional balance.
As for speed, if you check the results of last year's San Francisco race you'll see a couple of OC-1 paddlers were mixing it up in pretty good position with the surf ski racers. I think it's more a question of paddle used than boat; it's harder to paddle as fast with a single blade paddle than it is with a double-bladed paddle. With our Huki OC-2 using kayak paddles we can run as fast as the skis (assuming comparable paddlers). Even paddling this 45 lb, 24 ft boat by myself I have no problem running it in the mid to upper 6's and pushing over 7mph at will using a wing-blade paddle. I've raced it on an 19 mile race on Lake Michigan with a friend (an excellent canoe racer but not too experienced with kayak paddles), averaging 6.8 mph on the round-trip, including the stop where we had to get out of the boat and check-in. I'm just a mid-pack paddler at best, my partner was in a lot of pain the last three miles, and we came in 5th shortly after some elite surf ski paddlers, so I was quite happy with our placement. My wife and I have hit 9mph on flat water with this boat using kayak paddles, and my wife is a recreational paddler. Thus, I don't have any qualms saying an outrigger can have speed comparable to a ski. It's particularly beneficial having the extra stability when the conditions get messy.
Speaking of which, I'm ordering some extended iakos for our XS so I can use my kayak paddle as well as the canoe paddle (the ama is located closer to the hull on this boat, and it's harder to use a kayak paddle without bumping the ama); I'll let you know more about how this boat compares after that, since I know how I do in the V12 and I know my performance in the XS with an outrigger canoe paddle.