Some years ago I also left the airport during a long stop.
Back at the airport they made some real trouble because...they said it is forbidden to leave the airport while the luggage is still inside. Anyway, I am still alive.
And I would do it again. Foreigners have the 'no-understand'-bonus
Beside...there is a train station near the airport. 30NTD bus ride and there are some food locations. 15 Minutes to Taipei station by high speed train
Taoyuan is also not far away. 10...20 minutes by taxi, Taxis are rather cheap. Taiwanese food and local peaople can be enjoyed there. But never forget to write 'aiport' in Chinese letters for Taxi. And it is worth to try a Taiwanese McD or the other ones. Smoking is not allowed inside buildings. Beer is tasty.