I agree with the points that have been said - definitely put an expiration of 24-48 hours on every offer, or risk the other agent shopping your offer in the hopes someone else will submit a better one.
I understand your concern about the seller finally deciding to review offers after you retract it, but I would be very surprised if the seller hadn't seen your offer already. If the seller and agent are serious about selling the house, waiting for a better offer seems like the only reason (aside from some other unforeseen family circumstances) that they would delay responding.
Since you submitted your offer, every buyer's agent that has called on the house has heard, "Well we have an offer on the property, so any offers need to be best and final." Luckily for you, it sounds like nobody has submitted a better one yet, so draw a hard line and move on if you need to! If it doesn't work out now, it may come back to you later, and your negotiating position will be even better.