I brought this up a while back and was surprise at how many people were against me on it. Personally, I think it is a fair question to ask...
The one thing I do want to say is that I think it is ok when the beverage cart is in the way. If you have to go you have to go.
The problem is that even with the announcements that are made the FAs rarely enforce it. Like you said, a lot of times they don't even close the curtain. My main issue when I brought it up was that I needed to go and the Lav was occupies by a Y passenger. Yes, being in F doesn't make me have to go any more or less than anyone else. The fact is that the dedicated bathroom is a perk of F.
My concern is that once one person does it and gets away with it, a bunch of people start doing it. Generally, people consider being seated by a Lav a negative thing due to the foot traffic. That is essentially what F becomes in these rare instances.
On a side note, I've also seen the extreme of FAs enforcing the rules so I don't want to make it seem like it doesn't happen.