On a YOW-YYZ-SEA trip, I showed my NEXUS card to the agent at Pearson (no NEXUS if transitting through YYZ). He looked at it as if it was a dead fish I had just slapped on his desk. He wanted the passport.
You can still use Nexus in transit, not sure what issue you have had. If you do NOT have checked bags, you go out to the ticketing concourse, then right back in through the Nexus lanes as if you were originating in YYZ. If you have checked bags, you collect them from the belt and go under the tape at the far left of the cattle chutes for US INS. Let the agent directing traffic know you are going to use the Nexus kiosks and march past all the unwashed masses waiting for an agent. NEver been denied doing this.