If your goal is to avoid the hustle and bustle, then avoid the city proper altogether. Vancouver is nice, but a bit too busy for me; instead find a place in Burnaby, the tri-city area, Surrey or even North Vancouver. I haven't been to Toronto, but if I find Vancouver already too busy; my guess is that Toronto wouldn't fit the bill either. Again, that's not to say that either city is bad, just that if you want some peace and quiet, the downtown ain't it. Look at a place far enough from downtown that it's peaceful, but close enough that downtown is just a train ride away.
I'm within the Greater Vancouver Area, but definitely far from DT Vancouver... so much peaceful.