Quad loan DENIED due to zoning change

[Account Closed]

We are under contract for a quadplex which we were purchasing with a conventional loan. The quadplex is one building, with 2 units on each side and each side has its own address. This was legally built in the '80s, but since then the zoning has changed to R2. 

The loan was denied because of the zoning change, with the statement saying that If something happened to the building then the structure could not be rebuilt back to a quad.  I was told that we won't be able to get any institutional loan (now, or in the future) so we would have to do cash or seller financing.   Does this sound right? If so, I am assuming this must be a relatively common issue due to zoning changes. This may kill the deal for us, so I am wondering if anyone has any creative solutions on how to proceed?  This is our first multi family deal.