How to format a potential House Hack

[Lamontis Gardner]

So I planned to house hack what I thought was a duplex, but turns out that the units are listed as townhomes because both units have a different street numbers. Now the lenders are saying I would have to get two loans (1 as a primary and one as an investment property). Is there any way I can structure this deal that will minimize a 20% down payment on 1 unit or is this something I should maybe walk away from and find a true multifamily property?