A lot of very good points here, especially by Steven Kopstein who nails it on the head for how complicated NYC can be and why agents do earn their commissions.
Now I'm not a licensed agent but I do have quite a few anecdotes about how agents and others that work in the industry here put up huge roadblocks to maintain the status quo. The profits here are vast and any challenge to the establishment or efforts to create competition and transparency will be met by entrenched mafia-like backlash.
1. Last year I tried to FSBO a 2-family home that I own in Queens. I advertised that I would pay a 2.5% commission to any agent that brought a buyer. Truth of the matter was that buyers agents were pressured by their agencies to steer their clients away from my property if their clients asked about it. This drastically reduced my pool of potential buyers. And because NYC is a complicated market to operate within, buyers tend to rely on agents so they usually sign exclusive agreements to only purchase properties with that agent.
2. A very good friend of mine with a sizable real estate portfolio is also a licensed broker. Our common friend was looking for a $2.5m+ home in Long Island so my broker friend said he would help him find a home and represent him. The problem was the listing brokers did not want to work with my broker friend and were willing to let a qualified buyer walk. Why? The story that were stating is that they only co-broke with MLS brokers. That is absurd because it is perfectly legal to pay a licensed broker in NYS. Those agents in LI were simply placing their financial gains and maintaining status quo over their clients interests.
3. The title business in New York is also very scammy. Because lawyers don't get paid out sized commissions the way agents do, the unspoken secret is that lawyers direct title business to title companies who usually compensate lawyers in one form or another. While selling that 2-family house I mentioned above, I told my attorney that I would like to direct the title business to someone I knew. When the buyers agent found out about this they were willing to call off the entire deal. They actually scared the buyer by telling them that the title agency I wanted to direct the business to was not a solid title company. Pure hogwash but they did what they needed to do to keep the title business within their own circle. Kicksbacks? Of course. It's big business and many entrenched politicians have their hand in it. https://mobile.nytimes.com/2015/05/12/opinion/the-title-insurance-scam.html
4. Rentals. In NYC, The fee almost always is paid by the renter to the agent, not by the landlord. Many, many landlords with sizable portfolios are now able to hand out these juicy well paying commission jobs to friends and family without paying them out of their own pocket. Agents HATE no fee rental listings. What they have done to muddy the waters on this front is they post a huge amount of bogus ads in order to frustrate renters looking for no fee listings. As Steven Kopstein said most renters are very busy and once they waste a few evenings or a weekend because of several bogus listings, they will usually relent and end up renting a apartment with a fee.