I'm an internal wholesaler at one of the larger Life Insurance carriers in the space. Though we don't offer Whole Life contracts, I see them frequently, and I used to be contracted for a short time by another carrier who's bread and butter was Whole Life.
The "Be Your Own Banker" concept is something everyone is familiar with if you work in this space. Like most things, there are pundits on either side of the argument - most of them have an agenda because they're trying to sell you on one idea vs. another. Overall, it's proven to be a viable option, but like what many others have mentioned here there are alternatives.
The big thing to consider here is that every company offers a different product and pricing, and those products/pricing mechanism change very frequently. In my 3 years with my current carrier I've seen many many industry-wide changes due to the macroeconomic environment, attempt to take market share, attempt to hedge against certain risks, industry regulation, etc. This is why it's best to consult a financial advisor to see if your idea would work based on the current environment and your personal constraints.
In theory, if I could take a loan from my life insurance policy's cash value at a fixed 3.5% but then lend that money out to someone at 5%, I could profit on the spread. I would pay the loan back + interest into my life insurance contract and the excess would be up to my choosing. I could even put the excess into the life insurance contract too to continue over-funding it. This is where the power of compounding + potential dividends (for Whole Life contracts) would accelerate my values.
A lot of people who are high net worth & are heavily invested in Real Estate look towards Premium Financing to get these contracts started - but the details of this are a bit outside of my scope of knowledge.
Life Insurance can get a bad rep, but it's something every wealthy person uses one way or another.