This is what an AC agent stated about entering nexus versus passport when check-in. She stated one should use the passport, rather than nexus (as Carl Johnson said in his post):

Thank you for your email.

I'm sorry to hear that you have had some difficulties in using your Nexus card to check-in. Since October 22nd, our computers and check-in kiosks have been programmed to accept the Nexus card as a valid travel document for travel between the USA and Canada.

During the online check-in process you will only be able to enter one travel document, either your passport or the Nexus card. The system is not programmed to accept both. Since the Global Entry Kiosks can only accept a passport as a valid travel document, it is better if you enter your passport information at the time of checking-in, either online or at the airport.

I have copied some information below regarding the Global Entry program. Please see their website for more information.
It appears that the best way to make sure your reservation is linked to the Global Entry system is to always enter the passport number when checking in.

..........

Best Regards,
xxxxxxxxxxxxx
Customer Relations