The instruction stated it should go on form 4797, and the loss is subject to unallowable loss limit.

They remain distinct on Form 1040 with different tax treatmentOnly capital losses can offset capital gains (except for the $3,000 allowance)So even though material participation makes the loss non-passive, it's still an ordinary loss that cannot offset capital gains from stock sales on the return.The only way capital losses can become ordinary is upon the sale of your property, and that loss could offset your W2 salary.  

Without knowing specific financials, I would say cut your losses and sell.

For example, lets say I invested in something and it had $100k in passive losses.

Obtaining REP status does not release prior passive losses. 

You said that the coverage of the loss and I assume the cost of legal defense was an "exclusion" to title coverage.  

I have created a lot of loss in my taxes over the last few years and now want to get a mortgage for a new primary home.

Given the fact that if  there is a gain a future year the prior year loss can offset it, it seems that one would want to file the loss so they can offset that gain.

She took a major loss.

That's just gambling though.