I used to live in an older home with a similar problem, but instead of moss, I had mushrooms growing. It sounds to me that the moss is growing on a brick around a window, which would indicate to me that you have a sealant failure around the window, allowing moisture to enter. The brick itself doesn't really contain anything for plant life to sustain itself, but it will absorb a small amount of moisture. If rain is getting in and penetrating the wood around the window, and the sun is not able to dry the area out sufficiently, you will be seeing indications of water damage - plant growth is one of them. My advice is to look closely at the area around the brick for possible leaks and seal them up. Scraping the moss off will take care of the immediate moss problem, but not what is causing it in the first place. Be aware that water also causes mold, which could be harmful. I don't know if you have a brick facade with a 2x4 interior, but that could be a great hiding place for mold.

we have brick home. on the north exposure (the shaded side) the bricks are "green" where the downspout leaks and where the gutters have a tendency to drip or overflow. I discovered last year that a light solution of regular bleach mixed in water, in a spray bottle, sprayed lightly over the surface and then brushed and hour or so later with a heavy wire brush and hosed with heavy water spray did wonders for the green growth (it is gone) and did not appear to do any harm to the brick.