I know the feeling you're talking about, but just sit read your emails, text or talk to your companion. When I am riding in 'less than,' it's usually because all seats were booked in F or because it's an ongoing flight and for some reason, couldn't book F for that part. I DO wish I were sitting up there as I board, but it has nothing to do with who is sitting there.
Just relax, as the others have said, and don't worry about it, regardless of how you got to F, or how often you do. Haters will hate, no matter.