It could be sizing or it could just be the paste residue. Either way it will wash off with some elbow grease. I have always used TSP (tri-sodium phosphate) with great success. It's available at almost all home centers, hardware stores and paint stores. Just follow the directions on the box. I usually use a terry cloth towel to scrub the wall with.