Issues with purchasing second Primary Residence

[Yohannes Sium]

Hi All,

I had purchased a house last year as a primary residence with conventional loan and put 3% down. My goal was to utilize the primary residence and purchase a house every year, however I am having some issues which I want to clarify. My loan officer informed me that even though I have the D/I ratio to purchase another house, I will not be able to do so using the 3%. I have fulfilled my 1 Year commitment, however, he is indicated for me to utilize the 3% again I would have to move very far or have a drastic change which justify it.  Is this true? How do people manage to purchase a house every year utilizing the 3% down?