Yes you can drive in car, No problem
If you are going to Rainbow bridge, walk is also fine. you can see good views and night lights and holiday lighting. It is very beautiful.
I also have 4 months old kid, I left her with grandparents on canada side and Only me and my wife went in car.
My kid...