The reality is that by the time you retire, you will likely have a handful of different investments. This is good. Everyone inherently knows that diversification is good, but people don't often think of "product" or "tax" diversification.
You can look at a handful of options that are available with your 401k. Withdraw to buy RE, take a loan to buy RE, move to a self direct IRA to buy RE. You'll find that many of these options are not the most advantageous.
Your 401k was set up with IRS incentives to do what it's doing, sitting there growing until you're older. My best suggestion would be to find ways to continue to build up assets outside of your IRA to invest in RE, then just let your 401k sit. At most, I would consider taking a small loan from your 401k for a down payment, but even that makes the numbers hard to work.
Fast forward 10-20 years & hopefully you'll have a bunch of RE investments which can have a big impact on your taxes. This is when it could make more sense to start withdrawing from the IRA/401k, when your taxes are lower & more controlled.
I'm staring at a tax return from a RE investor who has ~$150k in taxable income for being a realtor. Plus income from 12 rental properties. Add everything up, and his taxable income is $633.
He has a total income of over $200k & his taxable income is only $633. Next to nothing! This is the true power of real estate investments. When you get to this point, you can withdraw from your IRA/401k virtually tax free. Patience is key...