My financial adviser’s plan was for me to purchase several properties and first trust deeds, both in and outside of a recently opened self-directed IRA.

The repayment terms for a 401k participant loan are equal monthly/quarterly payments of principal and interest (typically prime plus 1%) over a 5 year term (longer if used to acquire your principal residence).Please note that if you take a full $50,000 and then pay back the loan, you can't take another $50,000 until 12 months after the first loan was fully paid back.Per the loan offset rules that went into effect with the 2018 Tax and Job Act: if you leave your job and the loan is current at the time you leave your job but then the loan goes into default because you left your job, you will have until your tax return deadline (including any timely filed extension) to make the loan current by depositing the outstanding balance into an IRA (and thereby avoid the taxes and penalties that would otherwise apply).Alternative: Rollover Funds to A Solo 401kIf you are self-employed (i.e. active self-employment earned income separate from your w-2 income) with no full-time w-2 employees, you can set up a Solo 401k and then rollover your 401k funds once you leave your current job [NOTE: You generally can't rollover funds that you saved to your current employer plan until you quit.].You could then take a loan of up to 50% of the balance not to exceed $50,000.

A lot of forum members are very knowledgeable in their respective fields related to real estate investing, whether that is real estate sales, wholesaling, flipping, rentals, lending, self-directed IRA and Solo 401k investing, or tax and legal guidance.If you haven’t been to it already, you might want to check out the BP blog: https://www.biggerpockets.com/blog/The site has quite a few tools that can be helpful for new members.

My most complicated itemized deductions are my SEP IRA, HSA, and charitable deductions, but it is literally all numbers listed clearly on a ten page form I fill out, and it's literally plug and play for him. 

This could be your after-tax savings or, better yet, your retirement savings in a self-directed IRA or SD-401k if you have one.

Why not self direct IRA to invest in the RE? 

LAlso, where are you holding your cash (savings acct, checking acct, IRA, etc.)?

I also have 20k in a Roth IRA that I know I can touch no problem (if used for a first time house). 

This is a great forum with very knowledgeable members that will help to guide you in the right direction, whether that is real estate sales, wholesaling, flipping, rentals, lending, self-directed IRA and Solo 401k investing, or tax and legal guidance.

If you have an old 401(k) or IRA elsewhere, you may consider moving those funds to a Self directed IRA and then partnering with yourself on a deal.