There is useful information about Wisconsin security deposits refunds from page 8:
https://datcp.wi.gov/Documents/LT-LandlordTenantGuide497.pdf
I am not familiar with Wisconsin law by heart, am not a lawyer in any state, and the following is my general advice to anyone in your situation:
1. Assuming you followed the law, you should not have anything to worry about. I trust that you can back up all claims with proper evidence.
2. If you did not follow the law, the tenants may have a claim to their security deposit - but that should be it, as I did not see any penalties prescribed in the link above.
3. If the tenants decide to sue you - I mean, if they actually sue you, and regardless of either 1 or 2 above being the case - counter-sue them for the actual amount of your “damages.” That includes what sounds like breaking the lease, if I understood your post correctly. Of course, your case would be much stronger if you followed all laws in the first place.
(a) Your state may/may not allow additional (per diem) damages for the time that the property was out of service specifically to deal with the damages, but not if the tenants left before the expiration of the lease and you are already including that in your “damages” - no double-dipping. It sounds like you have a case where you directly lost a potential tenant due to the conditions of the property. Hopefully, you have something from that prospect in writing about their “disgust” with the property that you could show in court.
(b) Unless you are otherwise employed in “construction,” I do not see how you would be able to quantify your own hours of labor in terms of a dollar figure. But, assuming your state allows for this, add your labor cost. (This is where hiring a contractor is beneficial if you intend to go after your “damages” in court - you have an invoice with a dollar figure on it.)
(c) If you “overstate” the amount of your “damages,” you may p*ss off the judge - this part is not at all scientific.
4. Once in court the day of, there will typically be a chance for both of you to peacefully reconcile before going in front of the judge/commissioner. That is where the amount of your countersuit may give the tenants the final pause to settle with you - they may lose more than they expected to get back. Just mind 3.(c) above.
5. It is my understanding that it does not matter what - if anything - your lease says about conflict resolution between yourself and the tenant. I am talking “small claims” type of court, not a jury trial here.
But, as is often the case, they may be blowing smoke in your face with their threat to sue you. However, if you did not follow the law, their smoke may be toxic to the security deposit you are holding. Again, in my non-legal opinion, this would be the most they’d get back.
Also, you can try bringing up a suit first. If you win, you must consider the chances of collecting against these tenants - you know their employment situation, etc. If you cannot afford to speak with an attorney, the small claims’ clerk may be of assistance about court procedures - they cannot give legal advice, tho.
None of this is legal advice. You should always speak with an attorney for legal advice.