I disagree with the notion that brokers cost the consumer more money. There are obviously exceptions to this depending on the loan product but take rocket mortgage for example. They make a small spread on wholesale deals versus taxing the consumer outrageously on the retail channel. The wholesale channel is a lean business practice for the most part while with retail, there is significantly more expenses such as marketing, employee benefits, admin staff, etc. I sure as heck wish I had all of that!
On the other side of the coin, someone going through a broker or IMB to purchase an investment property, is likely going to spend significantly more than they would at a bank who holds the loan in house, because of LLPA's with Fannie/Freddie. Removing or lowering those LLPA's would level the playing field in terms of pricing, but making investment property loans more affordable is not going to help decrease prices.
I am not going to say there is a best lending model, but brokers legally can not be paid as much as IMB's, and I tend to be cheaper in more situation than not when comparing a loan estimate from an actual lender.