In fact, I still have a carry forward loss of somewhere around 100k on the books. 

You suddenly will realize that on papers you might be at no loss, but commissions (6% realtors fee) and blah blah type expenses will increase your loss or eat up your profit .

I'd like to sell my home and get something smaller, and then use the savings from the decreased mortgage to put towards investing, but that doesn't seem likely without a considerable loss.

Worse, what about unexpected personal losses not even related to real estate (job loss, health issues, catastrophic event, etc.)? 

Unless there were some special circumstances, paying par value for the note will cause general loss as future value depreciates. 

I let him stay in the homes he was working in, threw him money for food and gave him some cash when the house was done, but it was primarily room and board he worked for.

Can they get to food shopping easily?  

Some we have put into stocks, but we are really at a loss at how to invest the rest.  

But again you aren't flipping just food for thought.

Do you then sell for a loss or do you turn it into a rental?