My residence is also on 5 acre minimum parcels, but with the growth that is occurring in my area, I anticipate the zoning will be changing.

I have equity in my primary residents, so I plan to start with that.I owe about $124k and it should be worth 165 - 175.

I’m a US citizen/Canadian resident. 

The only real issue I've been seeing is that there is somewhat of a backlash from the community that currently resides there pre-gentrification.

Now hes not looking at this like any flippers on BP would, hes not trying to make a huge profit but more so save his residence and come out ahead with a slight discount. 

The original loan related expenses are not added to the basis of the property. the points that you paid on the first mortgages was deductible then because it was for the purchase of the primary residence.

They said because it's a rental property not my primary residence

Only way I can see this being allowed is since it's only a 2 story residence?

I have a plan to: a) Rehab my primary residence in Orange Countyb) Purchase a fixer-upper and have it rehabbed (relocation to either LA or Inland Empire).  

What you are implying beyond just getting mail at the property is that you would at some point sign a statement that you would be occupying the property as a primary residence.