Thanks for all the advice guys - very much appreciated!
My outdoor terminator does indeed not have a flap - just louvers - so I will install a new one with a flap. I also discovered that my HRV intake opening was gummed up and not allowing enough air into the house so the "negative pressure" thing you mentioned is likely true as well. Keeping it clear will hopefully stop the cold air from entering.
As to the physics argument, I'm unfortunatley a little ignorant, but can provide some additional colour. This started happening in the summer, when the weather outside was warm. It happens to be minus 30 Celcius at the moment in Ottawa, but the sweating was occuring in other seasons too.
I have bought insulation for the pipe, but am not sure if it will install it or not until I can figure out whether that is a useful step.