I imagine he would have to be the one qualifying for the loan, and my down payment would count as a gift to him, but I'm not very familiar with how this all works. 

All seemed okay, then they demanded I sign a gift affidavit for the funds.

If that is your first investing purchase have a contingency line item that will help to do the unexpected expense or it might be a gift. 

That is in the range of an allowable “gift”.

The reason he asked was there's a high end, and low end appraisal depending on what you need it for.Now, if it's intra-family, and you're getting it below market, to play it safe, do the sale based on the appraised amount, and have the difference handled as a gift.

I have recently been gifted my grandparents’ house.

Your time and efforts have value, and if you gift such value to the IRA by performing services, you are effectively making un-documented contributions to the IRA and artificially magnifying the tax sheltering, which is why this is prohibited.The alternatives in the NPN space are to joint venture with other parties or use some kind of a note broker who is doing the workouts.

The world gifted me with a job. 

Is it possible for her to"loan" me the money without incurring any tax penalties such as the gift tax?

We also send gifts to all of our new clients.