I agree with @Dan Heuschele in his last comment above. I understand that your part of the settlement must have been remote, with you not present due to travel. That only makes it MORE incumbent on the settlement agent to adhere to the written agreement terms, and account for the assignments of rents and prorates, etc. Not so sure about their responsibility to collect keys? That is MY JOB as a purchaser, at settlement, and I would make certain someone would represent me in person, in my absence, at closing, to get the keys.
Once the settlement occurred, essentially ALL sales contract terms "merge into the deed." Meaning, if I am recalling all of this right from Law School;) (Note Disclaimer: I am NOT an Atty or providing any legal advice here), that once the deed is transferred, the sales contract terms become irrelevant and probably not enforceable thereafter? Leases, yes. Sales contract terms, no.
Anyway, a good lesson to us all. I have made similar, more minor and luckily not problematic mistakes similar to yours, 2 months ago on the purchase of 9 units, so we all can make mistakes. I inherited 9 tenants and forgot to require an estoppel from the seller as to the deposits, rent roles, etc.
Oh yeah, another law school lesson I found useful and amusing, the "Call Girl principle." Once services are rendered, the chance of collecting fees or payment decline...greatly. LOL! Once you paid the seller, you lost your leverage over/with them to a great extent. They are taking full advantage of that error apparently. Was the seller a lawyer?
The tenancy you created thereafter, is completely separate from the rest.
I applaud you for maintaining a good attitude, planning to extricate yourself as quickly, safely and with the least amount of pain, and moving on with your life. Also your NOT doing anything that would have constituted an illegal, constructive eviction. Watch "Pacific Heights" again!