@Patrick Roberts thanks for your helpful insights. Would you mind answering my question at the end of this paragraph (2nd to last sentence) sorry I typed a lot of other stuff you don't need to read..
I never would have thought a conventional loan at a big bank would have such twisted fine print. just really disappointing that the loan officer knew we wanted to rent it out and when the real estate savvy guy wasn't there, made the other buyer sign something preventing that. she was rushed taking off from work, with the sellers and everyone there, and not in a position to read everything let alone negotiate a change of closing paperwork. they will lose a ton of interest on 5.5% loan if we refi with someone else, but it would likely be 7% so its a lose-lose, they will lose our business and it could be in their favor to just drop the abusive term, ridiculous that a conventional loan doesn't let you keep the place after moving out. never saw that coming and this type of "pulling a fast one" is exactly why the other buyer who couldn't be present on closing specified that the power of attorney for signing closing documents was limited to exclude anything unnecessary. This would clearly exclude a ridiculous term which is not included in other conventional doctor loans by default, especially given the understanding that we were investing in real estate and doing rental investing, not just buying a temporary place to stay. If this isn't clear enough, one could survey the other bank loan officers and see if the other doctor notes included this term around the year the loan was written, if this is one of only a few outliers which they threw the term in for, then any jury would likely conclude it was unnecessary and the power of attorney would make that void... but they as you say if the bank makes all the rules for their conventional since it's not "conforming..." then they could call the loan due and as you said the refi would be needed regardless. but there is case law that allows you to break some terms of a mortgage agreement if something is unenforceable, for example the sub-to community relies on the Supreme Court cases which say it's not required to notify the lender when you transfer the deed/title. This would also likely be unenforceable due to no authority to sign such an unnecessary piece of paper onbehalf of the remote buyer. Assuming the bank has the loan on their balance sheet (not resold), would they want to call it due if the interest rates only went up by 1.5% since the time of closing? they probably have a bunch of 3% nonconforming loans on their balance sheet so 5.5% is not bad, totally reliable payments, plus in a legal battle they'd likely find the power of attorney didn't extend to the ridiculous term and that there was no breach of any enforceable agreement.