Depending on your home country it could be more expensive to live in Canada. You must be willing to work, assuming you are getting spouse open work permit. Unless you have the financial resources, it is going to be difficult to manage a family on a part-time salary. Honestly, it's mostly financial, as the rent in Toronto is very expensive. Another thing is mostly to do with your children on a psychological basis, they will probably lose their childhood friends and will have to make friends, in a different country, and probably people who may not look like them. Also, the reverse will happen when you go back, unless you have plans of settling in Canada.
Let's not forget that it's a new experience for yourself as well, you may not be able to talk with your friends, depending on the time difference, and you will also have to make new friends as well. As someone who was born abroad, I have personal experience living in a country that is not my own, and hence I have strong opinions in that regard.