If this closet isn't in the basement and you don't have any other moisture problem areas then you might have a leaky roof. I had this happen to me in an old house I was renting. It had plaster walls that would just hold so much moisture that we never knew how badly the roof leaked in spots. We never saw actual water but a downstairs closet was horribly moist. Also, you might be getting moisture from underneath if you live on a slab or have a damp basement underneath.