I don't do holding agreements. When someone wants to rent from me, we sign the lease and I collect the first rent payment and the security deposit. Usually they are moving in the same or next day. If I am renting a unit that the current tenant has not move out of yet, I sign a lease with the future tenants and collect the security deposit. Then I collect the first rent payment when they are moving in. If any of these tenants decided they didn't want to move in, they are still responsible for the rent. I would offer them an option to buy their way out of the lease for an amount that would compensate me for the lost time of marketing and other hassles of getting a new lease signed.
As for whether you need to refund their money, I don't think you do. Check what your agreement says, you want to keep a fee, not a deposit, deposits are refundable. Since you said you would send a check, you should probably send one, decide how much you think you should keep for the time you kept the unit off the market for them. In the future, don't give in to yelling, and if you aren't sure what you want to do, tell them you will get back to them.