"Oh the customer insisted on buying the materials, it's not my fault"

I definitely would want the mail to go to the correct place.If someone wanted to sue you, the fact that you ignored changing your address would be a good argument that you are not maintaining corporate formalities, potentially having the protections of your LLC ignored.Most states have annual filing requirements for the LLCs so that would be a good time to update it.  

The long term goal is to do custom homes, starting with spec builds.

Consider if it will truly benefit your tax position, liability protection, or asset management, or if it will just add unnecessary paperwork.If you’re in a state like California, keep in mind that you’ll have to pay $800 annually to the Franchise Tax Board, file a periodic Statement of Information (SOI), and submit a Beneficial Ownership Information (BOI) report.It’s a good idea to consult a real estate attorney to ensure you’re making the best decision for your specific needs.

They offer flexibility in design and can be customized to include a business space.Shipping container homes: While trendy, these can be challenging for mixed-use (residential/commercial) purposes due to zoning and building code issues in many areas.

As soon as you start discussing your business practice and strategies with your customers (tenants), then all bets are off and you run the risk of Fair Housing violations. 

I'm looking for ways I can "protect" this private lender. 

While the landlord defined the dwelling unit as his single-family home; the court, after analyzing the statutory definitions of the term, the determined that the dwelling unit at issue was the separate room being rented to the tenant.[2] As a result, the landlord had converted his single unit dwelling into a multi-unit dwelling that was subject to Oakland’s Rent Adjustment Program since the individual rooms were not alienable separate from the title to any other dwelling unit.Under the court’s decision, tenants in Oakland renting rooms in a single-family home or a condominium are entitled to the protections provided by Oakland’s Rent Adjustment Program.

If its a very large property, you may also want to make sure you are marketing to any wedding, family reunion, corporate meeting/team building customers. 

This protects you in case the buyer defaults.Lastly, keep in mind that owner financing comes with some risk.