We personally have a rule of 3 in our office. We find that personalities are varying between all of our clients and we highly encourage our buyers to shop around and interview several people in each field. Especially when it comes in inspectors and mortgage or loan officers. Everyone is different and just because you hit it off with the broker/agent you may find a better fit with someone other than their "guy".
When it comes to working with the sellers agent keep in mind that at that point they represent you both, some brokers/agents are able to handle this very well, while others struggle with it. Remember that you want to have the best representation for you, someone who is looking out for your best interest. If they represent the seller too they are not always able to assist as much because they are what is considered dual agency.
As for red flags there are many, but follow your gut. You will always know what is right for you.