Are you dealing with exposed aggregate concrete? Gravel could be simple raked and turned over. If you are dealing with a solvent based spray paint on exposed aggregate, removal will be difficult. I think using TSP is your best bet rather than solvent cleaning. TSP should give you some scrub time and will eventually work out the paint. You might get a wire brush while you are at it.