It sounds as if there was either wallpaper painted over or latex over oil based paint and that could cause this problem. I would get a reasonable amount of the loose paint off and sand the edges and if you want to make it less noticable you can spackle the the edges. Then spot prime this area with an oil based primer and that will help hold it down, but there is no way to know if will not do this again. Hope this helps.

The paint I'm peeling off is like 2-3 layers & it leaves a noticeable hole. Wouldn't I have to put some polyfilla in the area or something to make it look level?

Yes you can fill the entire area if it is that deep but I would use drywall mud for that large of an area as it easier to use than spackle on large areas.