It sounds like there was a problem with the Post Office. Maybe they did not have a forwarding address because they also returned the package to the sender. Assuming you had left a forwarding address with the P.O., I would call the credit card company again and request that they waive the fees. It may help if you have had other cards with them.

The problem you may run into is that they do not want to waive the fees because they will not believe that the Post Office had a change of address but did not forward the mail. They may think it could happen once or twice but 3 or 4 times is improbable.

Good Luck.