I think it depends a lot on your experience paddling doubles too. I have very little experience in this area. I am as happy as a pig in mud while paddling an XT single or equivalent in a messy, cold 30 knot downwinder. However I'm pretty tense in anything other than calm water in my XT double with my partner who does not otherwise paddle. My partner is as relaxed as can be though because she has no idea how much I'm struggling. We've never fallen out so I think the ski is more stable than I give it credit for.
A mate of mine has a Nelo 600. He and I have taken that out a few times in various conditions and I am much more relaxed in that ski. No doubt it is partly due to paddling with another experienced paddler who just happens to have vast doubles experience too. The Nelo doesn't feel terribly slow but it's hard to see it being as fast as a ski so much longer and narrower.
Current boats
Epic V10L Ultra, Epic V9 Ultra, Carbonology Sport Boost X LV, Fenn Bluefin, Nelo 510, Fenn XT double, Nelo 600, Expedition Kayaks Azure, Mirage 732.
Previous boats
Spirit PRS, Fenn Swordfish, Fenn XT, Fenn Swordfish S, Think Zen, Epic V10L Club, Carbonology Sport Boost LV