I agree--I would not remove while the current tenant is there. Actually, I wouldn't remove at all (why pay the cost to remove them?).
As an insurance agent and a landlord with a place that has a fireplace in one unit and wood stove in the bottom unit (with shared chimney), I made it known that they were NOT to be used, UNLESS the tenant paid to have them cleaned and inspected annually AND they carried renters insurance.
It can vary by state, but check with your agent on how to set this up so that you are a named insured on their policy and it will cover your losses (including loss of rent) if they burn the place down (and you get a free remodel out of it!). Any competent insurance agent should know how to do this quickly and easily. And most of the companies I have seen DO charge SLIGHTLY more for having a fireplace and/or wood stove, but it is a small increase in most cases. And I don't recall seeing a policy denied for this.