There's two separate questions, here, and they have different answers. Downsides to closing now and refunding the AF? Yes. Downsides to waiting a while and closing after paying the AF? Not so much.

It depends if you ever want more cards from Barclay. If you keep this cards a few months, then cancel it, then wait a few months, you may be able to get another 60k for another AF.

If you cancel and ask for the AF refunded, don't expect them to take too kindly to you repeating that in a few months.

Always evaluate an action as to whether it'll make the bank "hate" you or not, in light of whether you might want to apply again for a card from that bank or not.