You can also get to Silvermine Bay/Mui Wo by bus, besides taxi. The "A35" bus is $14 each way ($23 on Sundays and public holidays) direct from the airport to Mui Wo, but it's quite infrequent, with a departure every 40-60 minutes. Or take a bus or taxi to Tung Chung town, and switch to bus #3 on Lantau to Mui Wo.
But I agree that it's a really poor choice of hotel to visit Po Lin Monastery.