If youi want a particular time, then ask for it on boarding and will likely be ahead of others
Whether someone is in the cabin through paying a 1st fare, upgraded using miles from business or has received an involuntary upgrade , that person should be treated equally to anyone else
No more reason to feel annoyed than a paid 1st passenger has grounds to be annoyed if a particular time is goine becuase some mileage upgrader selected a time before he did