you have just stated why infants are not allowed in an exit row.... this will prevent you and other passengers to leave th a/craft quickly and persons sitting in the exit row and required to assist crew with the evacuation of the a/craft... which you would not be able to... it is also an issue with oxygen masks as well.... the infant has to sit on the correct side of a/craft.

You must be able bodied and be able to open and lift door onto the seat or out onto the wing.

If the ground staff at the airport did not pcik up on this then the crew would move you as they are the final check before take off.