If you don't give reference letter, those years won't be counted. That means less points. I've heard of another friend who had the same issue. Even though employer refused to give, they said if NSNP officer contacts them, they can just tell that the person worked here.
But what is the problem with a big company like Infosys to give you an issue, if you have worked for them? I don't understand. Don't you know any old staffs, managers etc in Infosys?
Try your best, pray and see what happens. Then update NSNP the truth and provide as much proof as possible.
All the best. May be others can share their experience.