This has not been my experience at all; on most trips, I basically live in the lounge and am surprised how few fellow passengers ever leave their assigned spaces. Never been asked to leave, except for when we are nearing the final station and they are closing out the car. The first class lounge on the Starlight may change the dynamic, but that doesn't sound like it should be a problem based on the other posts.

In the worst case if you do feel pressured, I suppose you could migrate down to the standard class lounge for a bit - the view should be the same or better (slightly larger windows, I think; more than slightly pedestrian decor) - and then come back to the first class section. You'd be remiss to not have explored the entire train at some point over the course of the trip, anyway.