Well, the CRA is the only authority concerned with determining if you are or were a tax resident for 2018, 2019 and 2020 and now that we have this cleared let me.give you my 2 cents on the matter.
Given your situation, you aren't a tax resident and you didn't have to file for.any return, however, I would advice that you file taxes for those years even if its a zero return just to be on track of tax filing.( just an advice, you aren't legally entitled to do that).
When I was in your situation in the UAE too I phoned up the CRA and the agent made it clear to me that if I don't have any of the significant ties and just a couple of secondary ties that doesnt make me a tax resident, this was even confirmed when I approached H&R block to file my non tax resident tax year of 2019.
To make peace with yourself on that matter, try calling one of the tax preparation companies as they are a alot more knowledgeable of the matter and are easier to reach to via phone than the CRA.
(Try H&R block or Turbotax)