Hello @Brian Willard,
Great question, and to @Brett Deas's point.. they won't.
Technically the signed documents will be on record if you ever get audited for any reason and they can therefore prove that you acknowledged and agreed to those terms when taking on that financing. But the chances of this happening are very unlikely.
Obviously if you start buying 3, 4 or 5 "second homes" then underwriter's are going to start looking into this with a fine tooth comb and most likely find out if you have a history of getting rental income from these properties.
That being said, I am helping a client get his 3rd "second home" right now in Arkansas. One is a lakehouse near his primary where he likes to hunt, one is in a different state by his daughter who is off in college, and one is an underground home built into a burm so that they do not have the constant fear of tornados seeing as they are right in the middle of tornado alley.
With enough written letters of explanation, you can help the underwriters approve the loan if it all makes sense.
And again, @Brett Deas is correct. It is usually only 2 weeks out of the year you have to occupy the residence. Not 50%.
Let me know if this helps and if you have any more questions!