Is a non-functioning chimney really that big of a Deal???

[Melissa Gittens]

So my partner and I recently  got a property we found off market under contract and had an appraisal completed on the property. The property is a multi family that has two chimneys one of which has not been used for at least 10 years. In the appraisal,the appraiser made note of this non-functioning chimney being a safety hazard since some of the bricks around it are crumbling. Other than that the property appraised for well above asking. 

The property is being sold as is, and owner does not plan to make any repairs. We have conventional loan and honestly this is the first time I have seen anything like this before.  We got a few quotes from the various masonry companies its about $1,000 on average to fix, the chimney is structural sound and these estimates are to remove it completely.. We really want this property, should we pay the expense and continue with the deal? This is the only thing that is holding us up. We tried with the lender to have the appraiser remove this contingency with no luck.