My two cents as a two-penny wannabe lawyer are this. You have little to worry about. Of course being threatened with a lawsuit is worrisome. Of course actually being named in a lawsuit is even more worrisome. But IMO, they have no case.
For starters they have to prove damages. What damages??? At best, they "suffered" a delay. The "damages" in this cases are highly subjective because, nobody can predict and therefore prove the future. Anything could have happened. So how does one go about quantifying damages in a meaningful way when what those damages are, and if they are even valid in the first place, is subjective? Now, I'm not saying a clever lawyer can't figure out how to make, argue and win a case like that, but I do know clever lawyers cost a lost of money and they typically don't work for free or only if they win in these cases. Therefore, and as others have already pointed out, it will likely cost more money to litigate this than it will cost for any damages awarded (emphasis on the word awarded). And even if they push for legal fees as damages, again they have to actually win AND the court has to side in their favor of legal fees. That's a lot of gambling if you ask me.
Then lets think about their case. What case? Contracts serve both parties and are in place for exactly the reasons you are experiencing - it's actually a way to perform a good faith agreement, and the first example of good faith is the willingness to enter into a written binding contract. Contracts exist to avoid court, not as a tool to go to court, though of course they become an extremely significant piece of evidence once you get to court. Why?
If you end up in court, it's because two parties can't seem to resolve an issue among themselves. This is a service provided by our society, and is a burden on our tax dollars. Many courts are subsequently not keen on the idea of frivolous lawsuits, and often throw stuff out really fast for that very reason. Of course that all depends on the specific court.
Your contract, entered into by both parties in good faith, and my opinion concerning the validity of the contract is this, if you and the seller signed the contract, it's valid. The relationship either of you had with your respective agents is a separate agreement, which you will notice includes a separate contract. While the agents might get dragged into this as material players, and they are subject to over-sight and discipline from the regulatory environment that governs their license, any issue you or the seller have with your respective agents would need to be addressed as a separate case.
You stated your contract was one that was a) entered into by both parties (do you have a written contract signed by you and the seller? Do you have anything additional that reflects the actions made with respect the the repair issue that came up later?), b) included a repair contingency that provided the seller the option to repair any defects (don't do that ever again) and c) had a timeline in which the seller was to respond (time is of the essence, I'm assuming your agents all used the standard WB-11 or, any contract that was written by anyone competent will have that as a condition).
Seems to me it is as you say. The seller had 3 days to respond with respect to their decision to repair or not repair. This is really IMO where this one hinges. There is a difference between providing a response to if they are going to repair and responding to how or when they are going to repair. If they got back to you and said, "yes, we are going to make the repairs" AND you have that in writing, then they might have leg to stand on. If they did not give you an answer if they were going to exercise their option or not, then they did not honor the time is of the essence clause and you had the right to exercise your option to walk away.
If it were me, I would be reading that contract and comparing what the contract says to what actually happened and what can be proven. This is what the court is going to do too. DO NOT respond to any phone calls, DO NOT provide ANY information whatsoever to ANYONE. Tell that agent of yours you will SUE THEM and REPORT THEM if they provide your information to anyone. (You might not actually be able to sue the realtor, for the same reasons it will be difficult to sue you, but your threats will have exactly the same affect on the realtor that the threats being made against you will have, is they will think twice about it).
Based on what you've stated and my two-cent understanding of the law, I don't think they have a case. The agents didn't think so either as the earnest money was returned however, you mentioned some nonsense about the seller not having a contract with their agent??? Good grief. Maybe what is really happening is the seller is furious with their agent and they are mistakenly taking it out on you. People are stupid sometimes and react in silly ways, in fact it happens all the time.
Finally, after my first advice of do and say nothing, if you do receive paperwork from a lawyer, do not delay and take that paperwork straight to your lawyer and go from there. IMO, there is nothing to worry about until that happens and even if that does happen, I don't think there is anything to worry about, save this is going to cost a couple hundred in legal fees at that point.
Finally, this advice is based on what you have said here. I'm not trying to be a jerk, but frankly I don't trust that facts have been presented as they really are. I say this because people accidentally mess up the facts all the time, and it is in fact, the facts, that will ultimately determine the outcome. This is why the lawyers on here are advising the first thing you do is get a lawyer. They deal with people who can't get it all straight all day long, it's one big reason they're in business!
So, get your facts straight and move forward (and maybe that means do nothing) from there.
Good luck!
PS - Hey Chicago cousins, I'm not really feeling the love from a couple of those passive-aggressive insinuations that WI agents are all a bunch of dippys. If y'all want to have a discussion on all the unsubstantiated outrageous generalizations each of our regions has concerning the other, we can do that. But I warn you will lose the one where we talk about which region has a long and documented history of being corrupt. Love ya FIB's! LOL (Me mum's side is from Chicagoland too.)
