I came across the same thing recently. After steam stripping the wall paper, I had removed sections of this softer yellow colored sizing. I gave the wall a cleaning using DIF on a masonry sponge to remove all glue residuals and allowed the wall to dry. Then I used drywall compound to apply a thin skim coat and level the areas where the latex was removed. After sanding, I applied a spray texture so the smooth walls would match other painted surfaces in the house. Used Zinessers primer over the wall and texture, then painted as usual. I don't have any problems with the final result.