I removed the table - no-one seemed to mind. I can imagine that you might get a jobsworth crew member who would object, but I think you would be very unlucky. After all the table and the empty seat are there for your comfort and that of the person in the other seat. If you are travelling together and want to put the baby in there, it's not as if you're inconveniencing anyone else.