A few things don't make sense here, the crew trying to collect your tray 15 minutes after it was given out seems unlikely due to the amount of trays needed to be given out and the service of tea and coffee to everyone before any collecting in can start.
But let's just say it was before you had finished your meal and so your trays were left in front of you while others were collected.
Sometimes customers are leaning against the tray or have left their hand in a position where taking the tray would disturb them, also collecting a tray from the window seat can be difficult to reach.
Am I right in thinking that the crewmember tried to give you your breakfast tray while you dinner tray was on the table, if so I cannot for the life of me understand why thy did this, your dinner tray should be taken and then your breakfast tray presented?