Is it hard to rent SFR Primary and get a FHA on a multi?

[Matt Rusk]

Hey guys.  I am trying to figure out what is the next best step to scale my real estate profolio.  I started buying properties last July and now have 3.  Unfortunately I didn't really think of it as a business until recently once I started listening to all of the BP Podcasts.

My wife and I currently live in house #2.  I completely rebuilt the house and it appraised at 400k when we did our cash our refi.  Its a 1600 sq ft house that is as close to new construction as you can get now.  All mechanical, insulation, drywall etc are brand new.    Due to its location I feel it would be a great rental.  

Our plan was to rent it and buy a duplex with a FHA 3.5% Mortgage. I spoke to a mortgage broker and he told me it will be very difficult to get a FHA mortgage on a 2 family when I am currently living in a SFR. He says it will draw red flags or something. This was news to me. Is this the truth?

I am trying to put together all the facts and info so I can determine which is the right course of action.

Any advice would be appreciated.