The comment would probably annoy me too, but I doubt I'd complain. If I complained about everybody who upset me with careless remarks, I'd spend most of my day writing letters. Just mark him down as a baddy, you'll appreciate the nicer ones more!