Practically the US expects that you enter the US on a US passport and leave the US on a US passport. Right now, the former is easily enforced (incoming customs controls whether on US soil or preclearance) while the latter is not because the US generally does not have exit customs controls.
Right now I only have one passport (US), in a couple weeks I'll have my Canadian. Practically speaking, I don't generally need either passport to enter Canada or the US because I use NEXUS - that's the document (NEXUS Card) that I use in the machine at Preclearance re-entering the US, and in the NEXUS kiosk at Canadian customs entering Canada. Both the US Global Entry and CBSA NEXUS computers know that I have both citizenships (and will know I have both passports - I have to visit the office for them to add my Canadian passport). If I go by land and present my NEXUS card, they would see that document too. And by land, they're fine with an enhanced drivers license (issued by a few states, requires proof of US Citizenship, has an RFID chip for US CBP to read).
I don't know what will happen if the US implements proper exit controls, if the choice of travel document when checking in will matter. Practically at customs with my name & date of birth info they can reconcile my travel regardless of either passport I pick. But as far as the passport number that is transmitted by the airline to CBSA, I don't know if US or Canada cares
You can use your NEXUS card as the document that the kiosk reads at GE machines, but only at Canadian preclearance locations. I use my NEXUS card at GE preclearance because the bottom of the photo page (machine readable zone) is pretty beat up on my US passport and GE/airline kiosks have a tough time reading it. (When entering the US at non-preclearance locations, I tap the button on the screen to type my document number on the GE kiosk touchscreen instead). Assuming the US is aware of both citizenships this is fine, although I always carry my US passport anyways (in case I get sent to secondary) - if you get secondary inspection, generally US CBP and CBSA want to see your passport anyways. (The GE/NEXUS computers are separate from the ones that secondary officers use, and seeing the passport can help in verifying possession of the document and reviewing exit/entry history).