Hi, @Arianne L. - Time is money when purchasing real estate.
With that being said, has there been a pattern of poor or slow communication from this realtor(s)? Typically 24-48 hours is a reasonable response time. However, if there is a pattern of poor communication from this agent, then you certainly have the right to part ways, unless you've signed a contract to work exclusively with this team (some agents require this) then you need to address that before moving on.
I was in a similar situation when purchasing my recent income property. The realtor had great Yelp reviews and gave a fabulous initial consult. However, things went downhill from there -- his communications were not detailed, he provided no timelines (we were out-of-state investors), and it took him 48+ hours to respond to our most basic emails.
I made the difficult decision to part ways and it was the BEST decision we made. Our mortgage broker referred us to an excellent realtor who met our needs and efficiently got the job done. Hope things work out for you!