I have been booted out of my (even paid) Exit row seat on UA once as I was half asleep when the FA came around to ask if everyone is ok with seating in an emergency row. I asked "Excuse me? I did not get what you just said" and based on my German accent the lovely lady decided I was not fit to sit in the emergency row since she just decided that I do not understand/speak English.
Of course arguing that I just didn't get it as I was tired and not attentive didn't help.
Anyway, last few flights where I saw young kids or clearly disabled pax in emergency rows the FA swapped them out of their seats after the emergency briefing. And if they don't you should discretely point it out to them... did it myself once and it was no problem at all.