I made the mistake of not having my numbers right to move out quickly(I lose about $50 a month if I leave now. which isnt bad but its not good either).

A realtor focused on investment clients won't recommend a bad property management company or contractor because they will want to keep you happy. 

Then again most agents suck so you're going to go through a couple of bad ones first.

This could be good and bad.

You need a high credit score and no bad debt to move forward.

I was also looking into hard money lenders, however my concern is that I may not be able to refinance the property after the term due to my credit situation and I would be in a bad place with the lender.

If you're confident in your analysis, I wouldn't worry 1 second about the realtor's opinion good/bad.

A good or bad idea?

Success in business and investing is next to impossible with bad credit.

@Valerie Copeland yes, perhaps that is a "wiser" decision for the reasons you listed but I do not think you should be deterred from analyzing and (possibly) putting offers on properties strictly due to the fact that either it was no tenant or has a bad tenant.