I think the Irish border is a bit of a red herring tbh.
There is no reason why the UK could not carry on allowing free entry to EU citizens while at the same time imposing a visa requirement for employment and/or residence.
I don't think people were voting against tourism or business trips - rather it is the pressure on school places, hospital waiting times and jobs (especially lower-paid work) that motivated so many to support Leave...