First, stop worrying so much. Stress is #1 bad thing for the baby. It will be okay.
You're in Canada now. Having a baby with someone else isn't a crime. CIC will not deny you because you're pregnant with another man's child. In fact, it's none of their business until the baby is born! Even then, you just need to show proof the baby is Canadian and you're done. The fact you had a boyfriend while married isn't a crime either. Legally, you have done nothing wrong.
Your ex is trying to manipulate you and scare you so you keep him on your application. STOP all contact with him. All contact goes through your lawyer only. Make sure he's removed from your PR application and make sure you verify he is not on the application at landing.
Unless you lied (commited fraud) to get your PR somehow, there is nothing he can do to your application. If you were honest - then it's not his business. The application is between you and CIC. No one else. Now, CIC may receive a "poison pen letter" from him, accusing you of things. Just be prepared to show CIC that he's angry and trying to bully you. If you have any records of him threatening you, make sure to give a copy to your lawyer.
Basically:
- Stop all contact with him. Block all phone calls, emails, or forward to your lawyer
- Make sure he's removed from your application
- Nothing you have told us so far is reasons for you to be denied your PR
- Try to relax for your sake and the baby's sake