I've just managed to enter Canada being a spouse of a Canadian citizen, having a visitor visa and still waiting for PR.
Though it was written on the official government of Canada websites that immediate families of Canadian citizens do not need letters of authorization, but I can tell you for sure - get this letter if you intend to travel to Canada. This letter was checked in the city of departure, then in the city of a layover before taking the next flight to Canada and then in Canada at the arrival. Without this letter they wouldn't let me board.