I personally don't want to use them again mainly because of the lack of communication 

Obviously you tell this person that “only trusts you” to sell and pay the taxes and pocket $50k minimum more than selling to you.

nothing pointed at you personally but there has been build to rent projects were investors had to put up 50k non refundable and never got their house built.. so I can see putting it in escrow and releasing it upon closing for sure.. thats what i personally do with my new builds although since we average 25 to 75k of upgrades per house buyers have to stroke non refundable checks directly to us for those items.

Personally, paying off your mortgage early is a horrible idea.

Be careful of the advice you get from unqualified persons here or in any forum.First. 

Of course you can, just be aware of the potential consequences from a business perspective.Lots more to share but I'll spare you from reading a novel :) Feel free to reach out if you'd like to chat! 

We talked for several minutes, then out of the blue he said he would offer seller finance if he found the right person to sell to.

I make sure to ask that the person applicant gives me is the owner not the property manager 

I picked up another duplex for my personal portfolio that will bring me to 14 doors.Who do you use for property management?

I've recently started a small business, Alpine Property Rentals LLC, focusing on wholesaling and acquiring single-family homes for rental purposes.