Government assessments predict the biggest hit for regions that voted to Leave.