Seat assignments are not guaranteed. You did not reserve a specific seat, but rather a seat on the plane. If the FA asks you to move, and you agree, then I don't know that you can really complain about it.