There are caps on income, assistance amount, and restrictions on being a first time home buyer.

Conventional financing will offer the most competitive rates but can be more income restrictive and can be unscalable as investor portfolio expand.

If you want to purchase another home without fulfilling these criteria, VA will only guarantee your total outstanding loan amount up to the county loan limit of the home you wish to purchase next.There may be other restrictions such as how many you can have or how many in a single city.

This is particularly restrictive when attempting to qualify for larger balance home equity lines of credit of $350-400-500K. 

You would not be able to purchase an LTR as a second home because access would be restricted and the property would be subject to an occupancy agreement limiting use.Please be careful about listening to advice other than from a licensed loan officer. 

I've heard of restrictions for buildings on the historical register. 

Since ADU rules were liberalized by Seattle in 2019 (one attached + one detached allowed on any single-family lot, no owner-occupancy requirement, no restriction against separate sale). a number of spec builders in Seattle have been building what may look like a townhouse triplex but is legally a single family primary homes plus two ADUs, one attached and one detached. 

The drawback is more paperwork, guideline restrictions, and may require higher credit scores. 

Of course, with all things, the answers to all these matters will depend on the circumstances.California tends to have more laws on the books and requirements and restrictions that it can be a good idea to form a CA LLC for out of state property so that you as a CA resident are covered, and to try to have your contracts fall under the purview of CA courts.

@Nitin Gove check with the new builds what the guidelines and restrictions are for rentals.