They told me to send a regular letter to the following address
Ministère de l’Immigration, de la Diversité et de l’Inclusion
285, rue Notre-Dame, 4e étage
Montréal (Québec) H2Y 1T8
along with Name, DOB, File/Reference number, address, phone number and the reason for withdrawal.
I asked how long it would take, they said its not a standard process so there is no standard timeline, they also said that they may call me or send me an email to confirm before officially withdrawing.