Hello Cody. I am new to this forum but I did exactly that to help fund the down payment for a property and thought I would share some perspective. I went back and forth on whether to take a loan out against my 401k but talked myself into it. I convinced myself that I was only further diversifying my stock portfolio into something more stable. It was also something I could put my own energy in to see how big I could make it.
I was tired of my investment manager having all the fun but needed an initial investment to get started. The way I saw it is if I could focus on real estate and my investment manager focuses on the stock market, then at retirement, I should be ok if one investment strategy fell short as I had a hedge.
I agree that you are not getting taxed twice. I would also ask who would you rather be paying interest to. The bank or your retirement account.
The good news is that I guaranteed a 4% return on my $50k. Yes it came out of my pocket but It would still have had to come out of my pocket in some way as the money would not just magically appear no matter how hard I wished for it.
The bad news is if the market returned 10%, then I just lost 6%. The other thing to factor in is job stability. If you lose your job, then you still have to make the payments or take the remaining balance as a hardship withdraw which also has penalties you will pay. I also think there are some restrictions on being able to just take it as a hardship withdraw as well. However, if you got a loan from a bank, you will have to continue to make those as well.
The reality was that the stock market return went up 12% that year but my investment property made 14%. I also saw my property increase in value.
In the end, I weighed the risks and went in with a plan that my real estate venture might not make the returns the stock market would every year I have the loan out.
I told myself that this was going to be a one time investment to get me started.
I hope this helps!