I think it depends on the number of visitors from these countries, those who claim refugee status and those who remain illegal. For example, my country SVG had a great relationship with Canada until October 2012, now we are required to have a visa when visiting. Immigration has found out that criminals were able to change their names and get new passports in order to enter the country, others were making up lies and requesting refugee status in order to stay, also in small Caribbean islands the government can be very corrupted and do anything to get money. It has been found out a couple of people from other countries were able to "buy" passports from my island and others in order to visit countries like Canada where a visa is not necessary.
Just my two cents.