[Texas] Can I live in a 5 unit commercial MFH?

[Andrew Thomas Vedder]

Hello all. I've come across a 5 unit 1 bed 1 bath property that is commercially zoned. I spoke to lenders to try to acquire a loan on it but they said I cannot purchase a 5 unit+ and live in it as my primary residence. Their answer as to why? 

"It's just not done. Fannie and Freddie loans are generally for 1-4 unit properties." This blanket answer does not satisfy me but it is the same one I got from 3 different lenders.

I understand everything over 4 units is considered commercial, but is there any way I can use this 5 unit as my primary residence and rent out the other 4 units?

Could I rent out all 5 for a year then move into a unit once there's a vacancy?

Should I try working with different lenders that are closer to the location of the property instead of big bank lenders?

What are my options here?

Thanks in advance for your input.

-Andrew Vedder, Texas Resident.