It doesn't really bother me. Although an FA did once stop me from going to the forward lav to change my daughter's diaper, and the aisle was blocked with the drink cart so I couldn't go back. I really appreciated that one, as they hadn't stopped anyone before or after from doing the same. I guess in his mind, selectively enforcing that rule was more important than a stinky kid having to wait.
Point is, it needs to be enforced consistently or just removed as a rule. Either way, I don't really care, including when I'm in F and people come up to use the restroom. The only rule I really care about is that I don't really want a ton of people hanging out in the aisle waiting for it, but they do enforce that.