The only real issue I've been seeing is that there is somewhat of a backlash from the community that currently resides there pre-gentrification.

Now hes not looking at this like any flippers on BP would, hes not trying to make a huge profit but more so save his residence and come out ahead with a slight discount. 

The original loan related expenses are not added to the basis of the property. the points that you paid on the first mortgages was deductible then because it was for the purchase of the primary residence.

They said because it's a rental property not my primary residence

Only way I can see this being allowed is since it's only a 2 story residence?

I have a plan to: a) Rehab my primary residence in Orange Countyb) Purchase a fixer-upper and have it rehabbed (relocation to either LA or Inland Empire).  

I was in your shoes a couple months ago and after many midnight calls with my agent/investment buddy @David Ribardo, I finally closed on my first property last week.

What you are implying beyond just getting mail at the property is that you would at some point sign a statement that you would be occupying the property as a primary residence.

But all the hard money companies I've seen will not loan on primary residences..

We lease out a SF home in which current single tenant has resided for 4 years with minimal issues.