I apologize for the length of the post, I often don't post because the details matter and I don't have time to provide my full thoughts. Hopefully this is helpful...
First, congratulations on accessing Seller as an equity source. It is highly underutilized source of funding. Also, I'm impressed with your intent to realize that every change to a deal is an opportunity to negotiate.
I have quite a few notes that were held back by sellers, but haven't yet tried to access the equity I've realized via appreciation or "purchased equity", so judge my advice knowing I've never done this.
From my experience, information is power. I wouldn't share that you intend to refinance until absolutely necessary. I would start the conversation with both parties by asking if they need/want any or all of their capital back in the near future. That would allow you to identify their motivation if any. Assuming they don't have any motivation for a payoff, I would then (a different call/meeting) let them know that you are likely going to refinance, and ask what their intentions are with the money after you return it to them (at this point, you know they don't want their money back, and as other have said are probably pleased with how it is performing - 4.625%) I would possibly renegotiate the rate, and even possibly go up in rate if I could move one of the loans (or a partial loan) into secondary position on the other property. Depending on the relationship, you may be able to get a greater LTV on the single property than your bank/portfolio lender would let you go. This would allow you to ultimately yield greater cash-out than your initial plan, and likely better loan terms.
Another thing that would help the above scenario is if you were able to do partial payoffs (with discounts). If neither of them want you to pay them off fully, you could possibly pay them for example $20,000, but obtain a reduction in principal balance greater than that, say $22,000.
Ultimately, understand the goal (I think it is to access additional capital – approximately $56,000 if your portfolio lender will take you to 75%LTV), but don't get attached to the method. I also loved the idea to keep these people for future deals/funding. That is priceless.
Cheers,
Paul