It can be a minor irritation if you are trying to enjoy the peace and quiet and supposed exclusivity of the cabin. Especially if there is a queue. Can't say I've noticed it very often though, and most BA crew do send passengers back in my experience, certainly on aircraft with dividers and curtains.

I don't think the OP is DYKWIA for expressing annoyance. However I don't think it would have been appropriate to demand the passengers use another toilet. That would have been over the top. It's for the CC to police this. And as someone above has said, there may be instances where the trolley is blocking access to the rear toilet. If someone needs to go they need to go.