As others said, it doesn't even "need" to be fair in the sense that it is certainly not an unfair contractual term (in the legal sense of the term), but apart from anything else, bear in mind that obviously, your booking those seats took them out of inventory. Maybe if you hadn't selected them, someone else would have and would have paid for them, and maybe BA won't be able to sell them now. Maybe even some people will not have booked that flight because the seats they wanted were not available (you may think it is a far-fetched statement, but in my case, for instance, if I want to book CW or WT+ or WT and there is no more window seat available, I will try to book another flight (including on other airlines) if at all possible).

In that sense, while I understand that it is frustrating, I personally do not think that not refunding seat selection fees if you cancel your flight is actually unfair or even unreasonable.