fill your holes befor you sand, then you don't have to do it twice. What type of holes are you filling? Exterior spackle or caulk may be something to look at besides wood glue.
there is no wait between sanding and priming. Read the prime can to see what it recommends before you apply your paint. Usually the weather, humidity, heat etc will help to dictate a time frame. Usually it is safe to wait 4 hours-but again your primer may say you only need to.
Are you using latex or oil based?