You've talked to several far more senior members of this forum on this topic than myself, who have all given you more examples and explanations on this. My information comes from things like the link posted above and being on this forum for years and seeing this happen over and over again. Just like your agents told you you can work, there's been opposite cases where agents said people in your situation can't work.
From that link that was posted it says:
If you applied for a different kind of permit
- You cannot do any of the activities allowed by the original work permit. For example, you may have come to Canada as a worker and then applied for a study permit. If so, you must stop working once your work permit expires. After that, you cannot work or study until you get a new permit.
For example, way back when, when my wife was applying for her PR through the Experience Class, her original work permit was expiring so she applied for the so called bridging work permit. This bridging work permit essentially extends her existing work permit under all of the existing conditions and is the same as her work permit. She was allowed to continue working while on implied status from that application.
If you're going from a working holiday visa to an open work permit, you're switching work permit types, not just extending the original work permit.
Regardless, there seems to be some confusion specifically related to, "I don't want to extend my visitor status on top of it," - the reason why this is advised is simply because, if your OWP application gets returned (due to wrong fees, missing signatures, wrong info, whatever), then your implied status ends and you're out of status altogether. This goes beyond and regardless of any discussion on whether or not you can work on implied status and all that. If you're out of status due to the app being returned, then you can't reapply for said work permit until approval in principal of your PR application which happens at the very end of the process. Which means about 10 or so months of not being able to work.
This is what everyone that applies for the OWP has to think about (if their status is expiring soon) and decide whether or not they're gonna get that $100 peace of mind or not, basically. I personally didn't. I applied Feb 9, status expired April 14. My thinking at the time was, if they haven't returned my app in more than two months, they probably won't at all and it's fine. I got my work permit end of May. But I'm not sure if I'd apply that same logic here given the postal strikes going on.
So I just wanna clear up that we're talking about two different issues: whether or not you can work on implied status and whether or not you should extend the visitor status.
You are of course free to do as you will on any of these issues. We're all here simply sharing experiences and knowledge gained through years of dealing with Canadian Immigration.