There is a great deal of diversification within real estate that most don't acknowledge but is very real and then there is diversification of asset classes (stocks, bonds, real estate, private equity, etc).

You can do much better elsewhere.Now, if you have no intention of considering yourself a "Real Estate Investor" and you only purchase one property for diversification purposes, then paying it down might make you feel better and sleep better.

One common strategy in a diversification exchange scenario  where there is no debt is to actually purchase only one property to complete your exchange so you simplify the exchange. 

Next is deciding a market that will meet your desired returns and has characteristics like good job/population growth, job diversification, landlord/business friendly and a rising market.

As an example I am personally invested in Broadstone Net Lease, which provides instant diversification into hundreds of triple net leases across geography.  

Having more properties helps with diversification. 

This is commonly called a 'diversification' exchange.  

@Tom Shallcross summarized it very well. you also want to look at diversification of the employer industry. 

Anyways, my biggest concern is the lack of diversification.

The benefit is diversification.