The Captain was right to move you. I don't know what kind of brace you were wearing, but it doesn't really matter. The Federal Aviation Regulations (FARs) require that the Captain ensure that people seated in the exit row meet the requirements. One of those is that you must be able-bodied.

Is it possible you were ok to sit there? Sure. However, when the Captain is told "we have a pax with a brace of some sort in the exit row", then it is his ticket on the line if he lets you sit there. I am a pilot, and I would have made the same call that this Captain did.