"Gifting" a down payment?

[Brandon Allenczy]

I've heard of many people borrowing a down payment for buying a property when using traditional financing. Whenever I speak to a lender, however, they make it very clear that you can't "borrow" a down payment and it has to be "gifted" to you.  

Does this mean it's actually illegal to pay back someone who gifted the down payment? I'd love to hear from any lenders/brokers on this. Is this a big deal? Or is it one of those things that is "technically" not allowed but everyone does and lenders don't really care that much? 

I feel like it's kind of sketchy to say you're being gifted the down payment when you intend to pay it back but I'd love to hear other's thoughts about this.