I dive frequently in the summer (crayfishing and some spearfishing) and have plenty of experience entering and exiting through rocks and ledges. One has to be very careful because surge and swell are always more challenging that you expect, even on small days.
So the last thing I feel like is being caught out near rocks on a surfski. Unexpected surge and swell will quickly tip you out. Plus if your boat gets damaged out at sea you will be in trouble. Its just not worth it.
I've done several Barkers paddles where on a calm day my paddling companions have shot the gap. I've never been remotely interested in trying.