What kind of contract will you have with him? Owner/GC or Construction Manager?
This arrangement you have sounds like he is acting as a CM...essentially he is a liaison to the subs but carries no contracts with the subs (thats you). This method is great for cost savings (guessing he would be charging you at least 25-30% if this was a traditional build method). It just puts more work on your plate as the owner though it seems like you are ok with having a skin in the game.
Also, whose name will be on the building permit? I would push to have his name and insurance on the permit if you dont have a contract with him. He needs to be on the hook in some capacity legally, otherwise it could be easy for him to bail or not be accountable. If you had a bunch of experience in building homes I wouldn't be that concerned but this is your first build! Feel free to give PM me or give me a call if you have any other questions. I can give you some ideas about selecting/tracking finishes.