Backing out of purchasing a duplex last minute

[Landrin Lewien]

I need help! I planned to buy a duplex, everything was going good until 2 days before closing I found out that one of the tenants haven’t paid rent yet for this month, I did a final walkthrough and the place was very dirty, dog piss on the floor and smelled. I then realized that once I close on this duplex that I’m responsible for this tenant and responsible for kicking him out. We had a lot of discussions and I wanted to move closing back 1 month to make sure that the current tenant was out by then, (his lease was up in a month). The seller wanted to do everything he could to still close on the agreed day. But I wanted to back out because of the thing I saw on the walkthrough and about the rent not being paid. The seller then wrote me a big letter saying if we don’t close then he wants to take my earnest money, pay for a few septic issues that were fixed during closing and compensate him for rent payments on the other side because he served the other side notice to be out. He’s threatening to take me to court. Do I have a leg to stand on? The realtor makes it sound like I don’t.

Yes I wanted to back out but because of the issues that I saw on final walkthrough. Is that enough purpose to get sued? He really wants to close and he’s pissed off at me now. I’m scared, I need help. How should I handle this? I agreed to give him all of the earnest money but now he wants more. My last resort is court I don’t wanna do that.