They do take into account grammar, spelling and punctuation. Here is their assessment criteria.
http://takeielts.britishcouncil.org/sites/default/files/IELTS_task_2_Writing_band_descriptors.pdf
For band 5, for example, it says: may make noticeable errors in spelling and/or word formation that may cause some difficulty for the reader