Suin,
I think this maybe a case where the Visa Office Officer is in error. Occasionally(often) the officer in charge of a certain case is mistaken about a certain specific regulation as there are simply too many laws, regulations, policies, and procedures for a mere pen pusher to memorize.
The best approach is to send the Visa Office a letter stating that you have full custody of your kids along with the copy of the document(s) supporting your custody. Obviously, do not state that they are in error, simply restate your case.
If this does not work out, seek out help from an immigration lawyer to fight on your behalf. It should not be a lengthy nor difficult process.
Cheers and GL