Good grief, everyone's an MD! I always wipe down my armrests, tray, and seat belt. I don't seem to catch anything when traveling...I only catch the crap my family brings home to me when NOT traveling.
But if you only saw how black the wipes are after I've wiped down my seat area. It's pretty disgusting. I don't care so much about the cold and flu germs...those are in the air as well. I would just prefer not to ingest whatever nastiness is on that seat. Use your imagination. There are plenty of anecdotes on this site about nasty stuff going on during flights. Toenail clipping, babies getting diapers changed, nose picking. GROSS. I also wash my hands religiously and use hand sanitizer before eating outside the house.