Quebec is the answer if you wanna avoid english, but Im guessing if you're avoiding english then you most certainly are avoiding french. Unfortunately, your chances are super slim of being accepted since all universities in Canada require an english language score, with a total pf 6.5 and no band less than 6 for undergraduate, and total 7 with no band less than 6.5 for masters. I personally did my IELTS twice, I got below 6 in writing for a silly mistake of not doing it in paragraphs and then after repeating it I was able to get total of 7 with nothing lower than 6, but here's the thing that should worry you, even if you were able to find a college which doesn't look at your language (which almost they all do), you still have the study permit to worry about regarding this issue because they will not issue you a study permit knowing you aren't good at niether french or english. So my advice is, do your IELTS and even if you end up with a total of 5.5 or 6, just sign up for a course at the university which accepted you under that condition and go ahead and do it, its usually just one course which wont exceed a month and thats it. My friend is currently in Canada, Saint John, and he went with a total of 6 under that same condition. I've also seen people getting rejected under that term even if the university accepts you so its always better to just try to do your best and go on with it.
Best of luck!