Took me while to realise that there was a difference between swell and chop in downwind. I know you have qualified your statement by saying there was no wind.
As CDO said, I start to paddle hard when in the trough of the previous swell and get all I can outta the swell rising underneath me, then take some recovery stokes as it passes underneath me, sort of like interval training.
What I have enjoyed in the last couple of years is starting to recognise/learn the difference between swell and chop. Here in Western Australia we quite often get swell and chop separated by about 30 degrees. Once I realised that I could switch from the swell to the chop, zig zagging this hole downwind thing started to fall into place for me.
This technical aspect of downwind I guess is what keeps us old fella's
in touch with the younger paddlers for a lot longer, there is no substitute for wisdom and experience.