Tracy,
Sorry to hear you are going through this. Agree with other posters that if you have lost trust in this contractor, even if cost more, you need to cut your losses and find a new one.
As to what to do with the existing contact; depends on a number of different factors having to do with the size of the contractor, how big the contract is and emotionally just how upset you are about things.
If this is a small contractor then there are likely few if any assets in his company and it is generally very easy to shutdown one LLC and start another one with a similar name so suing them is just a money pit; its not fair but very common. This why vetting upfront is so important.
On the other hand, a larger contractor may have real assets in the company or even a reputation that they want to protect so they want to find away to avoid getting into legals which waste time and money for them.
With respect to contract size, in reality anything under 100K is not worth a full blown lawsuit as the legals will eat you alive even if you win; small claims court may be your best bet if you are OK managing the process or maybe using a paralegal.
On the emotional side, I strongly urge you to put that aside. This should be a business decision and a 'learning opportunity' about what to watch out for and what kind of people you want to deal with moving forward.
Oren