I believe manohman is correct and CBSA does not require Canadian citizens to obtain a passport to enroll in NEXUS, but only to present proof of citizenship and identity (such as birth certificate+driver's license)
I'm not sure about US Citizens in NEXUS. CBP is rather vague. Perhaps due to the legal requirements of US citizens to have a passport or passport card for international travel, they can't officially endorse enrolling someone in NEXUS without one of those documents.
LPR's of both countries may have other requirements.
Global Entry by itself does require valid passport, as the basis of the program is fast entry by GE kiosk in US airports using passport.