You can travel across the land border with just your NEXUS card.
But when flying, the answer is most likely no, for multiple reasons.
To enter the USA at the airport, you need your passport, the Global Entry machines require it, and any customs agent can/will ask you for it. Even if you can get through there without your passport, I think the gate agent can ask for it. Even if you get past THAT, you're likely to be asked for your passport when boarding the flight from the US back to Canada, and I've had my Nexus card rejected (either rightly or wrongly) in that situation multiple times before. Arriving back into Canada you could just use a NEXUS card.