Illegal multifamily... am I going to Jail!?

[Brandon Turner]

Ok - i'll admit, the subject line of this post is a bit sensational.  

I'm not going to jail. Not for this anyway. (But maybe for my killer dance moves.)

But here's the question i'll pose to the BiggerPockets crowd. 

(I'm working on a new book on Multifamily for BP, and I'm in the chapter on Zoning, so I thought it would be good to get some additional perspective!)

If I want to buy a duplex... and I find out that it's located in an area where the zoning only allows for single family (or, let's say, multifamily is maybe allowed BUT this duplex clearly was converted from a single family house years ago without permits) - obviously this is a "non-conforming property." 

So my question is: what are the downsides of this? Like... has anyone ever heard of a government forcing someone to turn their property back to a single family house after its been carved up? I assume most cities operate on a sort of "don't ask, don't tell" policy, but maybe that's just my experience.  Could a landlord who rents out his "illegal duplex" (i.e. an old house that has been converted) get in any kind of trouble, or would they ever be forced to fix it? 

I'll admit - i have several properties like this. I call em "monster houses." They can cash flow like crazy - but what are the legal / financial risks? 

Thanks all! Appreciate your thoughts as I put together this book!