What's the worst that can happen if I pull out of this deal??

[Archer Stone]

I'm moving to Florida with my wife & baby daughter. 

The market has been crazy. We really wanted a HOUSE with a backyard, but couldn't find one in our price range, so we made an offer on an APARTMENT instead.

Our offer got accepted. We did our inspection (which went fine). We waited several weeks for HOA approval. And today we finally got the approval! The tenant in the apartment is packing up to leave, and the closing is supposed to be in 6 days from today!

PLOT TWIST: TODAY we finally found a HOUSE, with a backyard, exactly where we want to be, next to a beautiful park with a lake and a playground for the baby! OMG.

The house is significantly better for our family and also it's a much better investment than the apartment. We just put an offer on the house and the seller said he will sign it tomorrow!!!

I want to pull out of the apartment deal and take the house instead, but I'm afraid the apartment seller will be angry and sue us.

I am assuming that the apartment seller (and his broker) will be PISSED that we wasted like 3 weeks of their time, they even fixed a bunch of little problems in the apartment, the tenant is moving out (though I think she will probably stay if we tell them that we're pulling out), etc. I know they're entitled to keep our $5,000 deposit, and I'm OK with that... as long as they don't SUE US and try to FORCE US to buy the apartment, when we already are buying the house.

Can they do that??? Would they do that??? What happens if we buy the house and they try to force us to buy the apartment?

Here's the language in the accepted offer (for the apartment, from which we want to pull out):