The Agent can force you to have some sort of inspection done to submit with your cancelation though if you have a licensed contractor do an inspection that may work as well.
I have had a property where the Buyer's lied and where playing games. Their Agent lied and so did their Lender and they weren't approved and the bank statements were altered to show funds when the Buyer's had none and were playing bate and switch. I forced the Buyer's to complete an inspection on the property as I knew the house would come through with an amazing clean inspection to then turn over to the new Buyer's. Though this was done in VA where contingencies are removed passively unlike CA where contingencies are only removed in writing.
Though it all comes down to the fact if you have a blanket inspection contingency in effect you can back out for any reason at all! You can back out of the escrow even if the inspection comes back perfect. It's your right as a Buyer and no Seller can force you to buy.
MJ
{Caveat to protect my license... this is based on info you mentioned and my opinion based on those facts. I do not know the whole situation and am not giving you legal advice.}