I've done a TPE turnaround twice, once with a 3.5hr layover and the other 1.5 hr. On the first one, there was simply too much time (they cancelled my outbound flight so I ended up with an earlier one) that I even took the train and checked out T2.
My experience is that the inbound flight is often early, and you ususally have more than enough time to even "visit" Taipei. I figured why not and got 2 extra stamps on my passport. I put "In Transit" on the landing form and no question was ever asked.
Could you do a turnaround instead and save on the cost of a hotel room? If you want to be absolutely safe, take the same plane on both ways (e.g. CX400/1, CX406/5, etc.). ^