@Orlando Goodon - I think you're confusing good debt with bad debt. This distinction was really illuminated for me when I read Rich Dad Poor Dad.
If your credit score has been at 580 (no shame in that btw - good for you for working to get it up!) then you've most likely taken on bad debt - debt on depreciating items (cars, credit cards, consumer purchases, etc...) Banks look at that debt very differently than debt (mortgages) on real, appreciating assets: houses, apartments, commercial buildings, etc...
And, if you're renting them out, they count a chunk of the rental income towards your overall income when calculating your credit worthiness.
To your earlier point, you don't want to be holding a $600k mortgage on a single family home when there's a correction. But a $200,000 mortgage on a duplex that throws off $500 a month? If you've put 5% down that's a $10k cash investment on your end, $500 x 12 = $6,000 a year in cash flow - so that's 60% Cash on cash return.
Even if that investment is under water (temporarily worth less than you paid for it) as long as you hold it and keep cash flowing, you're golden.