Apparently the latex paint previously applied was not done over primer. The base wood is rough sawn Cedar (horizontal siding). After scraping, and then using a wire brush, you can still pick and pull small sections of the old paint from the wood. Since it is rough sawn, the valleys retain the paint during scraping, and the wire brush does not really get to the old paint.
What would be the best approach to this problem short of sanding flat or replacing the siding. Which primer should be used on the bare wood, and to help adhere the remaining paint, (by remaining, I mean that paint that is not scraped and wire brushed)