As per my knowledge, Refugee claimants (those who applied inside Canada) don't need a reason to ask for their passports to be returned (this is based on experience). You can ask/apply for return of your passport as soon as you get your confirmation of PR (CoPR) and your new SIN is issued.
here...