The previous owners turned in into a single family home.

I've helped investors to sell fix and flips, multi-family homes, land, you name it!

With current interest rates, and they are at least 1% higher with potentially additional points for non owner occupied property, it is hard to find properties that will allow for positive cash flow after paying the mortgage, if you are looking for money for the down payment from a private lender, those are normally at higher than market and I am not sure it would be possible to pay it off.... please remember you need to take into account that you may not have a tenant for a certain amount of time, a tenant may stop paying rent, or you will have a big ticket item to fix, so I would wait if I were you and save to have enough for the down payment and closing costs and perhaps some money set aside for repairs. you also want to make sure that you have your own emergency fund outside of the money needed for the investment property because life happens ....

I live in Florida but my family is from Portsmouth and I just there for Labor Day.

If this was me, I would leverage my equity into a commercial office/retail/industrial multi-unit or multi-family property.

You want to talk to friends and family on this, I don't know anyone who would see this on the Internet and be like, yeah, this sounds like exactly what I am looking to do with my money.

Had some family live there last few years. 

Before listing the property, we were using it for ourselves and our family and friends.

I was thinking about buying the land (maybe it is 20-30k for the .25 acre lot), finding a builder in the area, and seeing if I could build a ~1,250 sq ft 3/2 single family house and see if I could make the math work. 

And later relisted as a "family compound" with most of the photos showing commercial aspects of the property removed.