Multiple Applications, first come first served or most qualified?

[Kelly N.]

I am considering changing my application acceptance policy from first come, first served, to choosing the most qualified applicant and wanted some thoughts on this.

1.  How long do you accept applications for?  Do you pick from everyone that submits before a certain date, or within a certain time of each other?

2. Do you go by what their applications say to rank them, go by gut feel or both, then run background checks?  I normally call references and run background checks on the first applicants to give me applications and fees, and move on to the next if the first one doesn't qualify.  If the first one does qualify, I return the app fees to any other parties- I feel like this is the right thing to do and would like to continue the practice.  

Kelly