The material you describe is an asphalt felt impregnated with a mineral surface made to look like brick. Also known as Insulbrick. It is usually handled as a non-asbestos material. It may contain asbestos, but any fiber is encapsulated by the asphalt during manufacture, and it is not likely released during normal removal. A bulk sample (small piece that includes the entire matrix felt and surface) can be taken to a local analytical lab for PLM analyis. That will cost about $25 to $35 and will answer your immediate question.