Buying other side of duplex as primary

[Scott Hibbert]

I bought my half of the duplex in Arvada a year ago.  Recently my neighbor mentioned he might move and asked me if I'm interested in buying his side.  Both are pretty much identical.  I bought my half with 5% down primary residence and rent the extra bedrooms.  I am now considering buying his half with another 5% down primary residence purchase and moving into his side.  My lender says the underwriter will most likely question why I am buying the other side as a primary and might ask me to put down 20% as it appears to be an investment.  I am curious if anyone else has heard of this or has any experience with primary residence purchases where the underwriter questioned the motives?

Thoughts?