This is a difficult scenario since latex is water based and brick is porous so the paint seeps right into the brick surface. PEEL AWAY 7 paint remover will do this job for you. It has the abilty to draw the paint out of the surface but it does need some agitation with a scrub brush and water