Buying a second house

[Eric Fernando]

I currently have a two-family in New Jersey. It has been exactly one year this August 2019, since I bought it. Bought it with conventional financing with 5 % down payment. I live in the lower unit and rent the upper unit.( house hacking) I am looking to buy another two family/ duplex in NJ. I spoke to a couple of lenders, and they tell me I would only be able to apply for an investment property loan and would need to put 20% down payment. They say that since my existing two family is in close vicinity of the proposed location of the two family that I am looking buy , they tell me that it is a big red flag for underwriters. Is that correct? Are there any ways I could get loan with a low down payment? Would it be any different if I had to buy a single family unit instead, and move into the single family and rent both units in my current two family unit? Would I be able to get a low down payment if I bought a single family? Would really appreciate if someone would be able to connect me with lenders, who have done this before