The only person you need to ask about a seat swap is the person in that seat. If there's nobody there, it's anybody's seat as long as they're entitled to it (e.g., no E- for E+ swap). You have no right to an empty seat, so I'd say you have nothing to complain about.