staining filler

My wife and I have spent many hours refinishing moulding for our Victorian house. Much of the woodwork had nail holes and large gouges. We used a stainable wood filler (I think it was from Elmers). We followed all of the instructions but when we went to re-stain the moulding, the filler did not nearly match the stain color. Any ideas about darkening these areas to look more like the stain color or at least not make them stand out so prominently? Any ideas would be much appreciated...Thanks.