Having problem with contractor again. please comment

[Nate Wong]

I hired a general contractor for replacing roof, remodel kitchen and bathroom along with painting and some other work. the whole project is about $ 65k which include materials and labor with a payment schedule . he's been very slow and for three months not getting much done due to he's having a few jobs at the same time . I was very upset with hes performance. they are on the job site for only couple of hours a day and sometimes not even there. Yesterday he tried to ask for $ 12k payment which I disagree due to his performance . According to the schedule of payment on the contract I provided to him , I basically do not need to pay yet . I have paid him $ 25k , and he said I need to pay him or he might stop the work and he said he used all $ 25k for materials and even he stops the work I have to right to complain to the contractor's board due to he has the receipts for the materials . My question is does he has the right for money even we have the contract with payment schedule saying t after what is done and pay how much?

what is the best to do if you were me?