Most wont recommend you travel for an extended period during this process unless you're a Canadian Citizen. Generally, short trips are ok, but if gone for several months, they will question your residency and potentially deny your application.
If you travel outside of Canada for an extended period of time, then they may not consider you as "living in" Canada.
You are required to live in Canada while family sponsorship is being processed.
Most wont recommend you travel for an extended period during this process unless you're a Canadian Citizen. Generally, short trips are ok, but if gone for several months, they will question your residency and potentially deny your application.
Who is eligible to sponsor their spouse, partner or child
You can sponsor your spouse, partner or dependent child if:
You’re at least 18 years old
You’re a Canadian citizen, a permanent resident of Canada, or a person registered in Canada as an Indian under the Canadian Indian Act
If you’re a Canadian citizen living outside Canada, you must show that you plan to live in Canada when the persons you want to sponsor become permanent residents.
You can’t sponsor someone if you’re a permanent resident living outside Canada.
Not only would it probably lead to your spousal sponsorship being refused if CIC realizes you have been out of Canada for 3-4 months it would also be a good idea to show that you are currently employed in Canada so you can show how you will support your wife and 3 kids when they arrive in Canada. Not sure why you would risk a refusal after having to reapply for your family after having two more children outside Canada after your wife and child had PR.
Depending on whether your application is considered being submitted in November 2018 or January 2019, since it was returned, looks like you have already left Canada once after you applied for their sponsorship.