First, read the contract you signed. If it does not have language concerning missed dates then you have no control. You must rely on smart and clever negotiations.
I bet the contract does have language allowing him to put a lien on your project if you don't pay.