I had the HH Visa card, got the bonus, got a few thousand points, liked my AMEX with more points per dollar better, cancelled the Visa, and they couldn't possibly and didn't take my points back.

Maybe you ought to ask the rep how long do you have to keep the Visa card so the points wouldn't be taken back?!