I agree - actually I set my OOO up about 45 minutes before I check out so that anyone sending me an email at the end of the day knows I am not going to get to it until I get back.
But the part that I struggle with is that trip home - is that eating into vacation time or is it still part of your "work" time?
Every case is different, but up thread you say that it has been a lot of Friday/Sunday, so I would consider it eating into your vacation time.
If this was a single case of arriving home late on a Friday before a vacation, I would let it slide and not make a big deal, but I think in your case it is different.
I would also have a joint talk with your scheduler and your boss/supervisor - which does not seem to be the same person. You really dont have a life at the moment it seems.