Try different compression/merging software as the results can vary a lot.
I've put my file through many applications and got the best results with PDFSam Basic. I had 3 full passports merged without a noticeable decrease in quality and a file size of just around 3 MB.
Then again, as the others said you don't really need the blanks. Up to you with what you feel comfortable with.