Most commonly, the issue with passports is the lack of stamps upon returning to Canada.
An exception is if you are a US citizen, in which case, when you travel to the US, then your EXIT from Canada is not stamped either.
I'm assuming your arrivals in other countries outside Canada have...