Yes, it's quite possible you may need to repaly the CCB for this entire period since your son is not living with you. Since he's not living with you, you are not entitled to the payments. At a minimum, you should notify CRA that he is no longer living with you and stop the payments immediately.
You have to notify CRA that you left 4 years ago. Notifying them now should at least prevent you from receiving any more incorrect payments that you will have to pay back. You will have to determine with CRA what you tax residency status was since you left, what you were entitled to in terms of benefits since you’ve been gone, etc. It looks much be better if you to contact them because eventually they will likely contact you wanting proof that you have been living and working in Canada for the past 4 years. Based on the fact you are posting it seems like you were aware that you weren’t entitled to CCB for one or both of your children.
Hasn't he been with your parents for several years? This is not classified as temporary and CCB should have been stopped.
