Like others have said, you should have put something there. For all they know, you were content with your aisle seat and had not a care in the world about the seat next to you.