After I left SL for work in Qatar, I have been visiting SL for my vacations a couple of times in a year and that is just for like 2-3 weeks only. Do I have to mention all these when applying?
Well I recommend you include all your travel dates when you apply for the PCC from Sri Lanka. They actually tell you to do this when you apply online.
I went to Sri Lanka a few times during my stay abroad so I still included the dates I was physically present is SL in my PCC application from time of birth to the day of the application.
So when I received my PCC, they actually mention the days of my visit, even though I was technically not living there any more (I'm still a SL citizen but they dont give a blanket PCC coverage date from the date of your birth to the day you apply). On the other hand other PCCs like the Indian PCC do that even though you might not have stayed in the country for many years (they dont specify the dates you were away from the country and came for visits etc.).
Anyways up to you, but I would include all the dates you came to visit SL, keeping in mind the stamps and dates stated in your passport. They will check those stamps and correlate the dates of your visits.