Question on sellers rights to back out of a real estate contract

[Logan Vierstra]

So I have a property that we just finished rehabbing and have listed on the market (Sioux Falls, SD) . We got into contract with an interested buyer and after the inspection he came back asking for us to do another 50k worth of work to the property. We said no. He started threating to get a lawyer involved claiming there were things we didn't disclose about the house, which is nonsense. 

Shouldn't we as the seller be able to back out of the contract at that point? 

My agent is saying we still technically have to negotiate with this guy, I'm just trying to get an understanding why. I understand that once he puts consideration down we are legally obligated to fulfill the contract. The part I don't understand is why we are still held to that contract if the guy presented us a counter after inspections. He's free to back out at that point if he chooses, why aren't we?