The Downsides!
The basics seem to be:
1. Traffic
2. Noise
3. Loo cleanliness (For me this is the biggy as US based airline crews rarely if ever clean the loo in flight no matter how messy it gets!)
4. A bumpy ride when the air is smooth
Whether you pay for F with $ or with your butt (frequent flying) you generally expect a better level of service, otherwise why pay? So Y should stick to Y and F to F for using the loos. Exceptions, of course, come up and should be dealt with in practical and reasonable manner.
My opinion? An occasional Y pax in the F loo is not an issue if there is a GOOD reason for it, i.e. blocked aisle. But a line in the back or sitting closer to the F loo is not justification and the FAs should be expected to enforce it.