In most states, if the seller has listed the property with an agent, they likely have signed an exclusive right to sell contract. These contracts historically transfer the entire right to sell the property to agent. And, there is usually a clause that specifies that if ANYONE who expresses interest in the property during the time of the listing period ends up purchasing the property, a commission is owed to the agent; even if the agent wasn't the one that introduced said buyer to the property (which is what seems to have happened here).
You may want to check and see what type of listing agreement the seller has with the agent and see if this is indeed the case.
However, if the seller reserved the right to sell the home in addition to the agent being able to sell the home, then you may have some leeway. It's usually called the "Exclusions" clause of the listing agreement. If this is the case, then you have a few more options.
It sounds like, in your case, the seller has delegated all sales issues over to the agent, since the agent called you after you contacted the seller. If the seller did sign an exclusive right to sell listing agreement, I'd suggest that you keep the home on your list and check on it again once the listing has expired. Others might have a different opinion. But, I've found that a litigious agent could come back and foul up any transaction. I've seen some agents put a lien on a home claiming a commission was owed. It caused lots of problems for everyone involved.
Find out what kind of contract the seller has with the agent before you make your move.