No, Silvermine Bay is not a base to explore other parts of Hong Kong either. It's basically good for two things:
- as a get away from the city; but the beach there has poor water quality;
- as a place to stay a night if you're going from the city to Po Lin Monastery via the ferry route. Not via MTR or the airport.
You really don't want to stay there to visit Hong Kong, as you have to take a ferry (at least 30 minutes each way on the faster more expensive ones, or an hour on the slower). Waste of time, and there's absolutely nothing to do at Silvermine Bay/Mui Wo at night.
If you're staying in Hong Kong at least 2 nights, I would really suggest you get a hotel in the city, and then take a day trip to Lantau. That way, it's a relaxing trip to Po Lin Monastery either via ferry or MTR, and you don't have to worry about luggage when you're over there.