Two things drive me crazy about seat moves:
1) When either the requesting pax or the FA simply assume I'll acquiesce (normally I am a bit particular about my seating preference).
2) When the pax are a couple or something and "want to sit together," make a fuss, then don't speak at all during the flight.

Having said that, one would hope the pax with medical needs would pre-arrange the appropriate seat, but I probably would have shrugged and taken 3 in this instance. It is of course, your own decision and you shouldn't feel guilty about it.