I'm looking to make this part of the retirement stream of income rather than an effort to accumulate multiple properties.

Due to item 1, adding an ADU can require years to start achieving any return (once the accumulated cash flow recovers the initial negative position).4) Adding an ADU is a slow process.

When a rental is taken out of service for personal use the accumulated depreciation goes away over a short period of time. 

Or do I just need to rent it out and accumulate more equity first?  

A smaller paid off portfolio seemed a lot easier to manage so that is the path I was on.. but I've come to the realization if I ever want to accumulate serious wealth I will have to use leverage and scale big. 

The vast majority of wealth accumulation is through appreciation, cash flow after rents increase, and amortization. 

Your responsibility is the DP, and the CF accumulates to pay you back the full amount of the DP. 

A lot of people get here and are looking to get bigger, but I think your pause is because you accumulated too much, too soon.

Over time, I’ve accumulated more cash reserves and don’t need as much flexibility, but I’m actually in the middle of a refinance right now using an interest-only mortgage—because the pricing with the lender and the corresponding ARM is more favorable than a comparable principal-and-interest loan.The key with these alternative or "exotic" mortgage types is that you need to be financially savvy and have a solid plan.

It may not appear this way on the surface but you are going to come out ahead by focusing on purchasing fewer but quality assets rather than becoming a door accumulator.