Avoiding 20% Down on Second Home

[Aaron Taulbee]

Couldn't find a thread that addressed this very clearly:

Currently live in my Primary Residence in suburb of Atlanta, GA. I'm interested in buying a lake property in N GA Mountains. In order to not put 20% down and keep more cash for rental down payments/repairs, is the possible scenario possible and above the board?

Buy Lakehouse with 3-5% down as it is intended to by Primary Residence. Switch licenses, billing addresses, etc to new Lakehouse Address while changing insurance on primary residence to make it a "second home".

In addition to saving on down payment percentage: It seems like if I "lived" in Lake House for 2 years, and moved back to current house for additional years before selling, I could sell either without paying taxes on gains.

What am I missing? How does the IRS officially determine where you live percentage wise when you have two homes within close proximity to each other? Before I consult a local Attorney/CPA, what are the legal and tax concerns with this type of planning?

Any help is greatly appreciated.