In my experience if you enter Canada with a Nexus card, CBSA never checks your passport. You get a piece of paper at the machine and then that's it. I did need my passport to get back to the USA though. Air Canada and United always asked me for it to check in, but I never tried just checking in online (I always had a bag). I always had it with me, but found it odd that no one ever really checked it. I presume that since Canada knows just about everything about everyone coming into the country (including who paid for the ticket - and even 20+ year old scrapes with the law - not me but others), they decided that was good enough for them.
I'd never travel internationally without my passport though. Just thought it was odd that no one in Canada seemed to even check if I had it.