You're right.
This is from the website :
Proof of completion of a secondary or post-secondary program conducted in French or English, either in Canada or abroad, includes:
•A diploma or transcripts from a secondary or post-secondary school indicating the successful completion of a program, in French or English, either in Canada or abroad. or
•A diploma or certificate from a secondary or post-secondary school indicating successful graduation from a program conducted in French or English, either in Canada or abroad.
Note: You must have successfully finished a secondary or post-secondary program in English or French. A single course done in English or French does not meet the requirement.
If the original document is in a language other than English or French then you must also include a letter from the academic institution indicating that the language of instruction was in English or French along with an official translation of the original document.
If I interpret that correctly, you only need to submit the letter from the school that English is the medium of instruction
IF your documents are not in English or French.
I just called CIC too, to confirm and the officer I talked to said that the diploma and TOR I have (which are in English) should meet the requirements and there should be no problem, and such letter is not needed, AT THE MINIMUM. So I can just hope that it is enough . I'm still gonna ask my former professor to help me out with a letter though, but I think I can safely submit my application with only the diploma and TOR