I had the same problem. I think this occurs most often when a water source is hooked up to a refrigerator but the refrigerator goes unused for a period of time which is often the case when buying a new home. I replcaed my water supply line that runs from the wall to the refrigerator. This fixed the problem for me. Mildew had started to grow inside the supply line. Also, I think plastic supply lines are more likely to have this problem. Try using a copper supply line.
I hope that helps.