Hi! This is more than likely caused by excessive moisture or humidity inside your house. The relative humidity should never be higher than 40% to prevent mold and mildew to grow and also for your health. Use a hygrometer (Bionaire sells a digital model that is very accurate and reliable) for about 20$, or you can buy the analog ones for about 5-8$, but they are not as accurate as digital models. A friend of mine had the same kind of problem; he hired a structural engineer and he discovered that the insulation in the walls was rotted due to absence of a proper vapor barrier. He also recommended that they install a mechanical air exchanger, which sucks moist, stale air out of the house and replaces it with fresh air from the outside. Excessive moisture can come from different sources, the most common ones are leaking roofing, or siding, especially where a vertical surface meets a horizontal one and around doors and windows and chimneys, a clothes dryer that does not vent to the outside, not using the exhaust fan when you take a bath or shower, hanging clothes inside for drying, cooking, or in my friend's case, because of the absence of a vapor barrier, the warm air from the house, in winter, mixed with the cold air from outside, condensed inside the walls, and eventually caused rot and mildew to develop in the insulation and on the wall studs.
Also, does the house have a real basement or just a crawl space? If so, the smell might be coming from there. Unfortunately, I can not give you info how to remedy to this situation.
In your case, I would start by using a hygrometer, and if the humidity level is higher than 40%, I would first do the simple things mentioned above like using the exhaust fan when you cook or when you shower, and if the problem does not disappear, then move on to the more complicated stuff. I hope this helps you!