Is my mortgage broker trying to scam us?

[Arpit Gandhi]

Background:

We bought an investment property in a major city in USA. Our agent is the broker and also a mortgage broker as well. We have enough funds to buy the property in cash but we decided to put 20% down and get a mortgage to make better use of our funds.

Events which transpired:

1. We put an offer for the investment property. We negotiated and the seller verbally approved our offer. While writing out the contract, our agent said he will ask the seller to give us a 2% seller assist and in return the broker would reduce his fees by 2%. Now, he wanted us to give him the 2% in cash so he saves money on taxes (Flag #1). The seller declined the 2% seller assist but accepted the offer we put on the property.
2. A week later, we use the inspection company he recommended (paid $624). The company came back with 7 recommendations. We also used his recommended construction company to give us an estimate. Being a DIY, I can roughly estimate the cost and my estimation was around $650-750. The company's estimate was over $2200. We took the report and the estimate and sent it to the seller. We were cc'ed in all the emails till this point.
3. 3 days later, our agent/broker comes back saying that the seller is only willing to give you $1500. So, we were like okay, so we can go back and ask for $2000 then. He said "No, the seller said it's take it or leave it". We were like ohh okay. Then, while ending the phone call, he said "Oh btw, the seller agent agreed on the 2% seller assist. So now, he will give you 2% during closing which you can give me in cash after." (Flag #2 - Until now, we were in all the emails and group messages but we were kept out of this. We feel like the broker made a deal with the seller/seller agent to get them to give us the 2% seller assist).
4. The $1500 was a 3 party check where we had to work with that construction company or we don't get the money. (Flag #3)
5. We accepted the $1500 and told the construction company to exclude few things.
6. Few days later, we asked the agent to send us the requirements for the mortgage. He sends us a whole list of requirements. We call him in the evening telling him that we have all the documents except for copy of the passport and SSN card. We can get that to you in the new few days. Our agent/broker said that he will start the process and we can submit the passport and SSN copy later.
7. We call him few days later and he tells us that he hasn't submit any of the paperwork yet and asked us to come in so we can sign the mortgage disclosure.
8. Few mins before our appointment, we get an email from our broker with our credit reports attached. My wife's is around 800. For me, 2 were in 780-790 range while TU was 705.
9. We walk into his office and he wants to be upfront with us so he tells us that the lender will look at the lowest number and your number is the lowest. I was like that's not possible. I use credit Karma, credit report provided by 2 different CC company and all 3 of them say it's 775-785 for the one in question. So he said that the lender won't give him much because of is. The "hit" would be too high for him to make any money off of this deal. So he said you have 2 options. Either take the higher interest rate or give him his share of the profit he would have made from this deal. We argued against the 705 number because that was inaccurate since all other sources were showing us 775-785 for TU. (Flag # 4 - We believe that he forged the number to get us to pay him the $2000 or take the higher interest rate).
10. I was multi tasking and checking my credit report to see how it can be 705. I ended up signing up for 2 different services (for free) and all showed the late payment (being disputed) and even with then, the score was around 775-785.
11. So after all this, I told him to exclude me from the mortgage. Just get it under my wife's name. Her income is sufficient enough to be able to get the loan for this with no questions. He thought for a while and then still came back saying "The hit means I don't make any profit so either we take the higher interest rate or pay him the profit he was going to make for a lower interest rate" (Flag #5 - Initially, he said my score was "low" and therefore we can't get a good interest rate but now the low score is no longer in the picture anymore)
12. At this point, we decided to get the best interest rate he can give us without us paying him a dime on the side and walked out telling him that we will reach out to other lenders to see what they can give us.
13. Now, the commitment letter is due in 8 days and the closing is in 23 days. (Flag #6 - We feel like he made us wait until now so we are pretty much bound to go through him to meet the time)

Since he's our agent as well, what do you guys think we should do? We already reached out to other lenders but we are not sure if they would be able to deliver on time.

Worst case scenario, we can just pay for this in cash and then do a cash out refinance later on.