You did the right thing coming here, the compiled knowledge on this site is better than any consultant.
You do not have to worry about your passport. It will be returned to you at your landing interview. Just to ease your mind, bring the stamped paper with the photocopy of your passport along with the seizure receipt to the interview.
You will not be withdrawing your case. As soon as you finish your interview you are automatically changed from a refugee to a PR. It is a very simple process. Basically they ask you to verify information such as the spelling of your name, date of birth, address, etc. They are very helpful, and will help you with getting your permanent SIN, updating medical, etc.