Hi,
Its so nice to see you are also living with him as i am also a citizen living with my husband but i did travel back and forth since 2019 but in your case i am sure when you gave birth to your child in Canada you also had baby passport with you but now you MUST send copy of your passport and your baby passport with entry stamps into Pakistan and shows that you want to travel with him as during pandemic its not safe to travel alone with a baby. its really nothing to worried about but IRCC is actually asked you to update them on everything that bring changes in your life. you do have a house in Canada to go back...please DO NOT CHANGE your address of Pakistan keep its as Canadian address because then you need to give proofs of going back to Canada.
ALSO ORDER NOTES and send an email with all the pics, passport stamps and baby details. I am also waiting like you here in Pakistan and its not an easy thing to do but it does shows strong marriage bond.
also apply your child NICOP Pakistan ID card with father name and shows Canada address as permanent address on NICOP.