When you should or shouldn't add units to a property

[Scott Harlow]

I have a 4-unit property on a large double lot in an area of town that is seeing excellent growth. I see this is a long term investment, but plans can always change. The back 1/2 of my property is undeveloped. I have plenty of room and the correct zoning to add 3-5 more units depending on the size of each individual unit. 

My only concern is this would take the property out of the conventional mortgage realm (more than 4 units) should I ever want to sell the property. Adding the units, based on my current cost estimates would make this property a cash flow machine, especially considering I would be building without adding any additional monthly debt - I will be self building/contracting a few units at a time with cash savings. 

Has anyone ever been in a similar situation? Did you regret turning your 4 unit into a 4+ unit property? Is there anything else I should be weighing here aside from cash flow and resale potentials?