When we lived in Montreal a couple years back we used to drive down to the city and be there within 6 (sometimes even 5.5 depending on traffic and border crossing). Try a smaller border crossing into the Eastern Townships and make the drive either west to Montreal or east to Quebec (which will be further, yes). We regularly went to Manhattan for the weekend and/or to New Haven to visit family (not quite as far obviously). Try Montreal, you can't go wrong with that city ... Stay in a B&B from airBnB or something like that and experience real Québécois culture - try Le Moulerie in Outremont if you like mussels! Otherwise book a room in Old Montreal and try the eateries there. It's like driving to Paris for the weekend