Are there any specific considerations for landlords?

What are some of the considerations or pitfalls to this strategy where we would start renting a single family residence out, then convert to our primary residence after ~5 years.

The considerations you mentioned are very important points indeed.

Questions for Consideration: - Is anyone familiar with Loan Direct LTD or James Baldwin?

Having been a corporate fraud investigator for many years, I say from considerable experience that your chance of getting much, if any, money back is almost nonexistent. 

One other consideration is take a look at the wording of the current lease they are on...

This is fair enough but when the income of the people transacting real estate has not changed to a considerable degree, it starts to feel unfair and you start to get push back. 

Or is it more due to the stereotypes around people on Section 8/the other tenants are paying a considerably lower amount every month?

I believe the play is to either improve the existing building and raise rents or the zoning allows for apartment buildings and the lot is considerably big, so maybe build new.

Tax benefits can help make a better business deal, but a bad business deal is hardly ever going to break even just because of tax benefits.The other big consideration you should have, in order to actually benefit from real estate fueled losses or credits, it will require a substantial investment of time - either to get over the material participation thresholds for short term rentals, or to get over the real estate professional thresholds for mid term /  traditional rentals.