I know many people who work with ITIN numbers as they have work visas and working towards getting permanent residency.

If the markets get better, they can refi to a lower, more permanent financing, if not, they can ride the wave or sell.It's all about your plan and anticipating what might happen so you can mitigate losses and assessing how much risk you are willing to take. 

eventually I’d like to build a permanent primary residence for my family on this land, and convert the first modular home we would rent into an ADU.

State law varies on the requirements for an installed feature to become permanent.

It is F at this time.

Here in NJ, unless you're buying As Is, the seller or buyer has to be able to obtain a permanent (not a temporary) CO for the property as a condition for selling or buying the property.So if you're NOT purchasing as is AND the municipality is requiring that you get rid of the illegal living unit in order to issue the CO, then the seller must do what is needed.

There are also opportunities to set up your own second units on acreage properties -- normally by moving a used manufactured home onto a permanent foundation.  

If you're going to be one of those that continually refinances your investment properties every 30 seconds to pull cash out, do 7/1 ARMs at the highest interest rate (yup I just said that) possible for the big fat lender credits (that's why I just said that). 

My wife was able to permanently quit her job to stay with the kids.

I have a feeling this won't be enough to wipe the smell away permanently.