This is a challenging situation. I agree with everything that was stated before. This is an unusual way to pay for materials and sends red flags from the get go. I think you are probably going to have to eat the costs at this point. But it may be worth it to take him to small claims court. That is usually the cheapest option, and often ends in the parties negotiating with a mediator. But if nothing was ever in writing, you really don't have a case. Just learn from it. That's all you can do.
And for me, the biggest lesson to takeaway is if someone is proposing an unusual arrangement, there is usually a reason why and to be wary. Sometimes people get so caught up in the "creative strategies" in real estate, that they look past why certain safe guards are put in place. Unless you are super experienced, I'd only enter into traditional agreements until you get the lay of the land. This payment method for materials is not traditional and doesn't appear in reputable books about flipping houses and dealing with contractors.
So keep educating yourself. And keep moving forward. This is just the cost of gaining experience and education. We've all been there in one way or another. So hang in there and don't be too hard on yourself. :)
Good luck!