I was able to downgrade my card to the no annual fee arrival version and keep my points, you lose any points that are not on your statement, so do this right after your statement hits.
my wife opened the arrival as well, so no need for me to pay the AF.