It must have been very disappointing to have been downgraded, I feel lucky it has never happened to me. When it does it is for operational reasons, and I suspect they may have been moving the cabin divider forward perhaps, hence why you could not board.
Whatever, the reason, BA is a very culturally diverse carrier, with staff and crew from just about every creed, colour and persuasion possible, so I feel you can be confident you were not singled out unfairly.