Wow! So much great feedback to you! I agree 100% with everything Mike M. typed to you. I would also add that you have nothing to worry about. I've been in this situation a couple of times on both sides of the table both personally and with clients. The super fact that nobody is mentioning here is this....if they sue you, the property is tied up! The cannot sell it until the lawsuit is over and the courts have decided if you are held to specific performance or let off the hook. This will cost them WAY more than any sane person should be willing to spend to get revenge in court.
Anyways, don't spend another minute thinking about anything unless the lawsuit is filed and you are served with a formal complaint. And never ever speak to their attorney about anything....do not let your number be given to the seller or their people.
I also have a thought on the whole agency thing. If they truly did not have an agency agreement with the listing broker, AND you didn't have a buyer's agency agreement with your agent's broker, then your agent also had no agency with you to even legally write the offer. In the abscense of a buyer agency agreement, the selling broker relies on the cooperative broker agreement based off of the listing broker's agency agreement with the seller (and legally works for the seller...not you). So then this whole thing is a mess and likely not a legal contract no matter what. The brokers can be sued for acting as an attorney since WI only allows them to fill out the legal contracts based on their agency agreements...it is otherwise acting as an attorney without a license. The real estate license is not a legal license...it's a license to simply fill in the blanks of WI state approved documents once there is a legal agency agreement between the parties. Also, the broker cannot legally take the earnest money without an agreement and cannot legally disburse it without a signed Cancellation and Mutual Release from the seller with instructions of amount and where it is being sent. Sounds like some brokers are in hot water to me!
I'm not an attorney, and these are just things I've been a party to. My 2 cents is to not sweat it unless it becomes a real thing.
Breathe!
Jacci