Hand fatigue. Any tips?

Hi guys. I'm wanting to pick your brain about hand fatigue. I finally got my Husky dialled in suspension wise I'm laying down some decent laps. Problem I'm having is fatigue in my hands on rough tracks. I used to be crippled with arm pump but since hitting the gym I only get this after a very long day riding a rough track and I also use the pro green arm pump spray which I think works so with the arm pump virtually cured the problem seems to be with my hands. I've got 3 laps in me before I can't hold on. I have quite small hands and I've tried to get a grip to suit me.

To be fair I am not the fittest guy in the world, my fitness is improving though as I said I'm hitting the gym mainly on cardio as I already have muscle mass. I'm weighing in at 16. 5st too (230lb in American money) so I'm not a light rider. I could stand to lose another stone of fat which Im on my way to doing. I am having to pull off track not even tired fitness wise.
The day before riding I eat clean with no fizzy drinks or alcohol (I mostly eat clean as my cheat day is Sunday evening after riding). The morning of my riding I have oats for breakfast with a pint of water then I have a banana on the way to the track. My food at the track is a wholemeal wrap with avocado, spinach, black beans, meat and feta cheese. I also have a few bananas throughout the day and I drink a load of water.

My suspension has been set up professionally and my forks converted to spring for a more plush feel. I run Renthal full diamond soft grips for their thin feel on pro taper bars.

I dunno if I'm death gripping but I'm always conscious of how much I'm gripping and try to relax my hands on braking. My knees are bruised from gripping the bike but I do sometimes forget to control my breathing. I ride at least once a week which I understand isn't enough but with a full time job and home life that's as good as it gets.

I'm racing next season so I really need to get this sorted as I have the speed but not the hand fitness.

Any tricks or tips I'd love to try. Thanks guys