Mortgage can't be sold on secondary market - issue?

[Eric Lee]

My partners and I recently got a multi-family house under contract. During the due diligence phase we discovered  a local ordinance on the parcel which states a multi-family property could not be rebuilt if destroyed. This prevents the mortgage from being sold on the secondary market. 

Our lender is willing to carry the primary note , but I am concerned about the long term viability of the property and the ability to sell, refinance, or how this affects the appreciation. I am a relatively new investor, but  I believe the government invests in nearly 90% of residential mortgages (unsure what percentage of that is on the secondary market) and I do not know how common it is for lenders to carry the primary note on a mortgage or a refinanced property. 

Has anyone else ran into situations like this and am I making too big of deal out of this?