If you have an old sidewalk and can break it up pretty small (less than 1ft chunks, more like big rock sizes), and you don't plan to put a different surface over anytime soon, the rubble is okay as long as you will be going a minimum of 4" over the rubble, preferably more.
I did the same thing, and put anywhere between 24 and about 8" and let it settle for about 3 years and there is nothing wrong with it.