I agree with Andy. Do your due diligence beforehand on who owns the note. If it is a large institution, you are most likely going to be fine.
I work in the financial markets full-time, and I see it as equivalent to a callable bond - a bond that the borrower has the right to retire at any time. Investors usually sell down the price of callable bonds because they are in the business of lending their money out, not holding cash that just inflates away over time. Plus, it takes time and money to reallocate those resources.
Bottom line is that lenders are obviously in the business of profiting off of interest over time. Why would they want to call the loan due when it is highly likely that they will now be turning a non-performing asset into a performing one? I would imagine if they even notice, they will run a credit check on you - so as long as your credit is decent, it's just business as usual.
However, moving your family into that house would be a risk I wouldn't personally take. I would want to hold the subject-to property as an investment instead.