Would they have continued the flight had the ground or cockpit crew realized that it was a panel that came off?

If yes, then obviously I don't have the first clue what constitutes true airworthiness, so I'll keep my trap shut and leave it to the highly-trained folks do their job.

If no, then I think the crews had an obligation to return to at least investigate the issue further before proceeding.

I'm certainly not claiming that their system of checks is lacking, but there is no perfect system so there's always room for improvement. Perhaps this is just one of those cases.