TRVs already have plenty of restrictions, what more would you have them add? A TRV lets you visit, you get no benefits, no healthcare, no nothing. It gives you physical presence, and so far as CIC is concerned, we (spousal applicants) want more than that already and they cannot guarantee that letting us have physical presence won't be abused. Being spouses gives us greater reason to try to take advantage, in their eyes, not less.
It's unfortunate, and it puts a real burden on relationships when spouses cannot visit, but at the same time, the more you ask that someone apply restrictions, the more likely that someone somewhere will start believing they are being "targeted" or "discriminated against", as if TRVs weren't enough of a target or discriminatory enough already, and start making a ruckus. Not to mention, as sure as God made little green apples, no matter how restrictive they tried to make it, there would be some who figure the rules don't apply to them and would try to take advantage. It happens.