On longer distance reserved trains, the remedy I've seen for an oversold train is to put people into the lounge car until seats free up down the line (usu. a stop or two).
Also, on some of the long distance routes, I've actually been assigned a particular seat by the coach attendant as I board - but this seems to be at the preference of the particular train crew and is the exception rather than the norm.