A small power sander can make quick work of the raised fiber on the cedar. I am in the process of painting the house and had a similar problem. The 4 x 4 porter cable hand sander knocked off the remaining paint and took down the sun worn fiber on redwood siding and trim.
I prefer to use an acrylic stain blocking primer, which is easier to spray apply and back-brush and cleanup, but the oil base should do fine if that is your preference. Paining over raised grain and fuzz will give an irregular finish to the paint. Dont forget to blow or brush off the dust after sanding.