Alot of good advice so far. There are a few components to sort through, so I thought I would just add a couple of thoughts...
Auction.com is here in AZ as well. They charge a $10K non-refundable deposit for any winning bids. I have been to alot of them, and have gone through the bidding process and have completed the sale. All that just to say I have experience with Auction.com and they seem to be a pretty reputable business...at least here in AZ.
I DO know that on all of their contracts, within the "fine print", there is a reinstatement clause which covers their butt in the event that the owner has somehow reinstated their loan with the lender, or filed bankruptcy to halt the sale, or SOMETHING to legally halt the sale. It's not exclusive the auction.com. It's that way with all trustee sales.
Regarding the agent/helper that assisted you....was he an actual Realtor? Many of these trustee sale guys aren't licensed, but they go to these things all the time and know the system, so they offer assistance to people who aren't comfortable with the process. They get paid by the Buyer for their assistance. Sometimes that is in the way of a flat fee for their time and research beforehand, as well as their expertise in bidding and advising you in how high to bid. Many of them require payment for that help regardless of whether your purchase comes to fruition or not. The rationale is that if they spent the few hours at the auction with you and the time beforehand to research properties, that time is valuable and should be paid for, regardless of the outcome that is out of their control (being outbid, etc).
IF this person IS a licensed Realtor...WHOA! This is super illegal. My guess is that 1 call to his broker will solve this problem right away. Between the betrayal the broker will feel in his wallet, and the liability that this agent has brought on to the broker's business, I would be SHOCKED if you had to go any further than the broker as it will be HIS business impacted by an legal action, along with the agent.
Now, on a side note, you mentioned that you signed some type of agreement with this guy beforehand. I would want to know what those terms were to accurately determine whether or not he is entitled to any funds from you. The agreement should accurately define how he would be entitled to a fee and/or commission from you. Make sure you have that in front of you when you call the broker.
Please don't let this experience sour you toward Realtors OR Trustee sales. It sounds like you found a bad seed, but in general, Realtors are a great tool and resource to guide you through this process. As a 20+ year Realtor here in AZ, it breaks my heart when I hear about these kinds of experiences. That is never how it should go.
Best of luck to you!