Try soaking it with some hot water and scraping it with a wallpaper scraper. It should be removed as you want the thinset to bond directly to the cement floor, not a veneer of something over top of the floor.