I am by no means an expert - far from it. I have a small ranch with wooden bulhead doors to the basement, 4 steps down to inner door (again wood) with glass pains (this is an old door and by no means as insulating as new doors). I can't imagine not having this door - keeps the basement much warmer.. I've been in this house for 13 years and the doors were there before I got here. I see no problems - however I do have cinderblock and not solid concrete so maybe that makes a difference.

I can't imagine the temperature difference or the speed of temperature changes to damage your walls. Unless you run a kiln or blast furnace in the basement.

I would be careful about trapping moisture. Maybe a fiberglass or metal door instead of wood.