Both passports should be registered at a NEXUS office. From there, you can use the NEXUS card at Global Entry kiosks @ Canadian airports with US Customs Preclearance. If you return to the US directly or at a Canadian airport without preclearance, then use your US passport in the kiosk.
When I added my Canadian citizenship certificate & passport to my NEXUS profile, I was reminded on both trips to the NEXUS office (one trip for each) that I needed to enter and leave the US on my US passport. At YYZ it didn't matter because both GE and Nexus kiosks there will read the NEXUS card itself (and both passports are associated in the backend).
I always select my US passport when picking a passport on airlines. I flew WestJet once (which allows you to pick the NEXUS card instead) and tried it. Online Check-In and customs at the destination was fine, departure gate agent at LGA would not allow me to board without showing a passport.
This is actually not completely correct. The NEXUS card itself is actually a completely valid document on its own for air entry into the United States or Canada.
US 8 CFR PART 235—INSPECTION OF PERSONS APPLYING FOR ADMISSION § 235.1 Scope of examination.
(e) NEXUS Air Program Participants.United States citizens, Canadian citizens, and permanent residents of Canada who are traveling as participants in the NEXUS Air program, may present, in lieu of a passport, a valid NEXUS Air membership card when using a NEXUS Air kiosk prior to entering the United States.
US CBP - What is NEXUS?
Under the Western Hemisphere Travel Initiative (WHTI), the NEXUS card has been approved as an alternative to the passport for air travel into the United States for U.S. and Canadian citizens. However, if you are traveling without a passport we advise that you contact the airline first. You will also not be able to use the Global Entry kiosk upon entering the U.S.
Even the last sentence
isn't universally true:
Can I use my NEXUS card at the Global Entry kiosk?
The NEXUS card will be recognized by the Global Entry kiosks in Canada Preclearance locations only. When using the Global Entry kiosks in Canada Preclearance locations you can use your passport, U.S. Lawful Permanent Resident card or NEXUS card. You cannot use your NEXUS card at Global Entry kiosks outside of Canadian Preclearance locations. At all other ports, you will need to use your passport or U.S. Lawful Permanent Resident card.
Canadian Border Services Agency concurs:
Can I use my NEXUS card as proof of identification and citizenship in non-NEXUS lanes when entering Canada?
Yes. If you are a Canadian or U.S. citizen, and a NEXUS member, you may use your membership card as proof of identification and citizenship when entering Canada by land, air (when coming from the U.S.) or boat when using non-NEXUS lanes.
That being said, good luck checking in with any non-Canadian airline with the NEXUS card, even with Canadian airlines a lot of the gate agents at the US points of departure won't recognize it as a valid document, and customs (both American and Canadian) will generally expect to see the passport if at secondary (or at airports like YUL where everyone using GE kiosks gets at least a basic interview). Also, you need to carry any supporting documentation for your purpose of entry into the US or Canada anyways, although this isn't needed if you're a dual citizen.