Wholesaler still wants to be "middle man"

[N/A N/A]

I have signed a contract to purchase a residential property. The contract was assigned to me by a local wholesaler. Now that I am getting ready to schedule appraisals etc. my lawyer is advising me to speak directly with the owner to set it up. His concern is that the wholesaler does not have the right to give me permission to enter the property.

In a nutshell, the wholesaler is insisting that he remain the middle man for everything, including supplying payoff info etc. because he is concerned that I am trying to go around him and boot him from the deal.

Is this commonplace, or should I insist upon direct contact with the owner?

Thanks for your help!