The water in Taipei is safe enough. Just on general principles of water that is "foreign" to your system, you should avoid gulping it from the tap, but for tea, soup, teeth-brushing etc., it's fine. I'm a little more distrustful of water in some of the smaller towns. Every 7-11 (there's one on every block) will have a stupendous variety of water for sale at modest prices.