I am travelling with my wife(Open work Permit), child(Study permit) and infant(Visitor Visa). My wife doesn't has job offer, but they are travelling with me.
Hard to say in that case.
You can definitely travel. Not possible to give a concrete answer for the rest of your family. Your travel is clearly essential. Not so clear for the rest of your family.
We've seen some people here successfully board. We have seen others refused and their travel deemed non-essential (this would apply to your wife and potentially your children). Generally speaking, those holding OWPs are allowed to board if they have a job in Canada. With just an OWP (but no job in Canada), it's a dice roll and CBSA is sometimes saying there is no reason for this person to be traveling to Canada right now.
There is a permision letter you can request from IRCC. However it's taking a long time for people to get responses and we have seen IRCC refuse to issue the permission letter in cases such as yours.
So short answer is that it's unfortunately hard to say what will happen.