I'm no expert in this but you MAY need to have the morter re-pointed. It is a job that requires some skill especially if you're not use to working w/it, or if it's aestetically important to the 'look' of the house.
The morter does crumble over time. Mother Nature is to blame. However, just because it's crumbling when you touch it, that doesn't necessarily mean you have to 'fix' it. If the morter is falling out in 'chunks' or pieces, all by itself (w/o touching it), then it's definitely time to have it repaired. If it's staying in place but 'comes loose' when you touch it, it's hard to say if it needs fixing. I'll lean towards the NO for an answer because LOTS of 'things' are fine, until you touch them.
The Brick Info Folks has a Residential link w/a FAQ and a way to ask ?'s. Try them if a Mason isn't of any help.
My best to ya and hope this helps.
Jay J -Moderator