No, you will not lose it when you are with a Canadian citizen. When it is time to renew your PR card, you will have to fill out how many days you were away from Canada in the last 5 years, if it is more than 1095 days (3 years), you will have to fill out another part of the application explaining your being away and one of the exceptions is that you were accompanying a Canadian citizen which you are so no worries.
