jam40 - You could have and should have recharged the card. One can get all its value back + HK$43 of the deposit when you're done. [Full HK$50 only if the card has been issued for 90 days or more.]
But you are right that each person needs to have his/her own card.