They have undertaken boarding by seat number a couple of times for my domestic flights at T5 without any mention that Gold, Silver, OW Emerald etc can board at their leisure so I just do the same as you and take no notice. The 3rd party security people who scan the BP first to check your photo don't seem to care.

I have never had a problem yet but if any BA staff at the gate challenged me, I would just do the same as you and politely explain the benefit of priority boarding, whilst making sure I took a note of their name so I could send in a complaint later and ask for them to receive some further (re)training on the rules and benefits of the EC/OW alliance.