Pulling out initial down payment...taxed?

[Dustin Beam]

Hello beautiful people of BP,

So, I'm no CPA, but it would seem that if I do not apply my original down payment on a property towards the new purchase, it would not be taxable as it wasn't actually a gain. So if I originally bought a $100k property w/ 20% down, I should be able to pull out that $20k after the sale and not be taxed on it. 

Anyone know if that's kosher with our friends at the IRS?