When you live in that duplex for 2 years and then sell you'll be able to take 50% of the profit tax free (the allocation for the side you lived in). 

And you don’t have to pay it back when you sell, you do an exchange.

this is a direct relation to lack of inventory.. new construction on many markets is as robust as its ever been.. we have had spectacular years the last 3 years of course during covid was tough with supply chains but my 500k house I am now selling for 700k and my build cost have gone up by maybe 100k.. so record profits for us.. 

The lender does charge the flipper 4 points upfront, and the note represents 55% of the property’s ARV, so I’m not too worried about needing to foreclose if it comes to that.The lender uses his own capital for initial funding and then sells off the notes after that.

The other challenge is selling it, there is a smaller buyer pool for hospitality in a small market but that was known going in Did you work with any real estate professionals (agents, lenders, etc.) that you'd recommend to others?

It will make the loans more valuable if you ever decide to sell them and decreases the chances of default. 

The options are, demo the lot, split the lot and then sell 2 lots to the builder or build myself.

Quote from @Sarah Ali: The seller has a contract with the agent, which means they can't legally sell the property without compensating the agent.

I told her no and then she told me to sell my crypto, sell my stocks, apply for credit cards and offered me financing.

I think it would be better just to stay on good terms and let him know you will contact him when you are ready to sell.