Portsmouth has only two ferry operators using the facilities , Brittany Ferries and Condor.
They don't have a dedicated freight terminal. Crossing times are long, fuel is expensive, so cost of crossing is high, as BF passengers know only too well.
Dover on the other hand has a dedicated freight/cargo terminal. Crossing times are much shorter so much cheaper.
Ferry operators using Dover are DFDS Dover>Calais, DFDS Dover >Dunkirk, Irish Ferries Dover >Calais and P & O dover >Calais
In addition, the road distance to Calais for fresh food exports from the Netherlands, Belgium, and Northern France is shorter than to Caen, St Malo, Le Havre.
Nothing to do with Brexit, a lot to do with crossing times, cost and freight handling facilities.