I own a nice home and pay a mortgage every month.

They are not complaining at this point about having to make a mortgage payment and pay rent to me, so perhaps I am more uncomfortable than they are, but how do other landlords deal with this risk, or does it just come with the territory?

I think it is related to the high property taxes, high cost of living in NJ in general, the new SALT law and creeping mortgage interest rates...those who don't buy with big profit margins built in will be challenged to make money when they exit the deal. 

Because rents went down, and have not come back up to 2007 levels, my rents fully occupied match almost exactly with my mortgage payments.

OR Should I forget about paying off current mortgage, use the $85k down payment towards buying town home?   

I would actually consult with a mortgage lender to make sure you can or have a path to getting pre qualified for a mortgage.

With this type of loan I would find a good mortgage broker who has access to multiple loan types and just ask them if there is a clause or what they know about it.

You'd still have to put up some of your own capital and give up some upside however you'd take massive risk off the table.Another option; get a 50% LTV 1st Mortgage and use the proceeds to buy more properties that generate income day one.Land development is a high stakes game. 

Mortgage Lending11.

I was a Personal Banker before that which gave me some good exposure to lending, primarily doing refinances and referring new purchases or larger deals to a mortgage partner.