Hi
As other posters have mentioned, biomechanics issues can be the main problem. Look up Lordosis of the lower spine on google. Check the pictures and compare with yourself. That tilt of the pelvis with a tight ankle and hamstring can lead to severe lower back pain when sitting for extended periods.
There are two things that you can try, active mobility for the hamstrings and ankles (not stretching it is mostly useless) for mobility look up Dr Kelly Starrett, he has a some amazing books on the subject and one is focussed on watermen, so has a lot of relevant advice for paddlers. The link to that book is here
Waterman book
The second thing is a lot of paddlers assume because of paddling that their core is strong. This is actually not the case. You have to specifically work the core and in different plains of movement. The one place most paddlers are weak is in anti rotation. I know what you are thinking, we rotate all the time why would we want to be strong in anti rotation? The reason is the rebound effect, if your core is strong against anti rotation, when you rotate and slightly relax you will find your body will go back to neutral instinctively and then you rotate to the other side. Over thousands of repetitions in a paddling workout, this slight relaxation of the core to neutral stabilisation makes a huge difference.
You can look up anti rotation exercises online but my favourite is the
Pallof press
Get strong in this movement pattern combined with the mobility exercises and you will be able to paddle as far as you want to paddle. It may take a while but I have seen decent results in as little as 3 weeks.
Sorry for the long post but I hope it will help you out.
M