(beginner) Bought 1st home, wondering about investing in a 2nd

[Jordan Sugar]

Hi! I am interested in investing in real estate and I just discovered the podcast and website. I'm looking for a bit if guidance. 

My wife and I are in our early 30s and live in the LA area with 2 kids. We bought our first house last year. We got a 15 year mortgage, with 20% down, but we are currently on a plan to pay it off in about 5-6 years. 

We had talked about purchasing investment properties after we paid off our house, but I am wondering if we should start sooner. I am reading a Robert Irwin book which suggests buying a home every 1-2 years as your primary residence, and renting out the previous residences as investments. However, my wife loves our house and wants to be here for a long time. 

All this being said, does it make sense for us to look for investments now? We are currently paying more than double our mortgage to get rid of this debt, but we could slow that down to save for a down payment on another house. However, my understanding is that if the investment property is not our primary residence, the financing will not be as attractive. 

Any suggestions? Thanks so much!  

Jordan