Use of self-directed 401k after 59-1/2 to build primary residence

[Owkaye Go]

I'm 61 years old and I've had a Fidelity 401k for the past 7 years or so. I also have a Traditional IRA. I do not have anything "Roth".

I'm self-employed on a part-time basis with no other income. I've been tax-deferring nearly all of my small income each year since 2010, and I'm planning to build a new home and move into it next year.  At the moment I've accumulated about $150k in retirement funds.

I've read that with a truly self-directed solo 401k I can use some (or all?) of my retirement funds to invest in my primary residence.  So does it make sense to replace my Fidelity 401k with a truly self-directed checkbook-control solo 401k, and then use those funds to finance the construction of my new primary residence?

My thought is to use these existing retirement funds instead of getting a construction loan. Then my primary home will be paid off -- free and clear -- so my only housing costs going into retirement will be maintenance, taxes, and insurance.

Bottom line:

Is there a better approach for me to accomplish my goal of building and moving into a new home by the end of next year? Or does this plan make as much sense as any other options I might have?