Hey joezimmer,
Travelling is no problem as you are applying outland - you are free to travel as much as you want during the process, so no worries.
If you were applying as common-law it might have been difficult to prove that you have lived 12 months without interruption, but if you are married then there is no issue. As long as your wife qualifies for sponsorship and you have sent proofs of your relationship, your marriage certificate and explain why you spend so much time outside, then it should be fine.
Chances of interviews for US/Can couples are very slim, so most likely you won't have to go to a VO.
AS you are from a visa exempt, they won't ask for your passport anymore (since december 2011), they will ask for the photocopies and issue the COPR on a separate sheet, so it won't be a problem for you to travel.
Good luck with your application,
Sweden