Right, but since they're living with me (a Canadian) in the US, there should be no issue with the permanent residency status of my wife and kids.
source: https://ircc.canada.ca/english/helpcentre/answer.asp?qnum=1466&top=10
You travel with a spouse or common-law partner
Your spouse or...