If you have a valid US visa, you can just enter into US for a couple hours or so and then just return back to Canada.
However, if you do not want to enter into US, just tell the American Border Service Agent that you have a US visa but you are here for 'Flagpole'.
It should not affect your US...