Certain types of molds can and will eat away wooden structures over time. Certain molds are more aggressive and if the conditions are conducive to continued growth damage can spread rapidly to other locations. Wood is a food source and mold will colonize on any surface that will allow it to feed.

Since mold spores are microscopic, they can travel and colonized almost anywhere.

So, not only is it possible for your side to become infected, it is also possible for other scenarios including health issues.

So, if you smell mold on your side, there is a good chance there are spores.