@Brent Shields - sorry about the late reply, didn't get the notification (or overlooked it).
Backlinks are important. They are definitely a ranking factor. However, not the only one of course. I think that some SEO folks get too obsessed with their pursuit of backlinks above all else and their efforts become spammy (hence the penalization like you mentioned).
There's a difference in a backlink strategy for local SEO vs national (like e-commerce or SAAS content marketing). Correct me if I'm wrong, but I'm going to assume that you and most people on this forum are interested in local SEO. I have a few comments to make about backlinks in response to your question:
1) I tend to look at backlinks as a positive side effect of A) writing valuable content that people want to link to, and B) effective PR. And that's 90% of my backlinking strategy right there. In terms of local SEO, your most valuable and relevant backlinks are going to come from other websites and publications that are tied to your area. The exception being that a link from Forbes or HuffingtonPost is going to be helpful regardless. So with exception to the big publications, which links from would be great, my local efforts would be focused on earning links from other local publications. Write guest articles. Do interviews. Offer to add value to other people's publications and the links will come. Link swapping works within reason, but don't over-do it. For example, a page of "recommended local partners" would be a great way to exchange links with other professionals in your local network.
2) In regards to penalization... it's not something you should worry about UNLESS you hire a company to purchase you backlinks, or utilize a Private Blog Network (PBN) to link to your sites. Those are the kinds of tactics that lead to problems like getting flagged / deindexed by Google. So just don't do that and you'll be fine. However, a local car mechanic linking to your real estate investment webpage is fine. You used the term "unrelated sites"... and that can defined liberally or conservatively, but in a nutshell, just don't pay for links and you'll be fine.
3) Backlinks are too often prioritized over other SEO strategies. A few for example that I would priortize (for local sites) over soliciting backlinks would be:
A) Google Reviews - This is one of the easiest and most powerful way to build your local SEO rank. It's the lowest hanging fruit, yet so few people grab from the tree. Google Reviews, Google Reviews, Google Reviews! An incredible SEO factor for local SEO.
B) Press - Like I mentioned earlier, write guest posts for local online publications that educate their readers. Don't take that as to write ADVERTISEMENTS because that's not going to fly. Write legitimate guest posts. It's fine if you have to widen your audience a little for the guest posts. For example, if you only catered to investors, I don't see a problem in taking your investor experience and writing an article geared towards regular home buyers if that's a bigger audience for the publication you're writing for. A link is a link. Just be sure to mention in the article that you work with investor clients and are taking a few lessons you've learned from investors over to help home buyers. (Just an example, I don't know what you do).
Another form of press is to get interviewed. You can even help craft the interview questions for the publication! Just another form of press. In the time it takes to solicit people for backlinks, you could put the same energy towards building your brand, getting PR, and earning backlinks at the same time. I don't understand why people solicit backlinks instead of going the PR route. It's like killing two birds with one stone.
C) Optimize your website with basic SEO fundamentals. Proper meta titles and descriptions, H1 tags on every page, navigation, internal linking, sitemaps, submission to Google Search Console, etc. You have no idea how many folks I come across who are out building backlinks and haven't even done this step. Their backlinks are kind of worthless if Google doesn't understand what your site does and what to rank you for. Sometimes taking care of the SEO fundamentals are all you need in certain local markets because there isn't much competition. This + Google Reviews = Success for A LOT of local businesses.
D) Web Directories. This basically just means listing your website places where it's available to you like Google Business Listing, Yelp (although I hate Yelp and don't recommend if you don't already use it), Facebook, BiggerPockets, YellowPages, FindLaw, WeddingWire, etc, etc, etc, depending on your industry. My caveat to this is to not go around creating a million profiles JUST for the ability to post your link. It's not worth it. Better to have a one or two very strong profiles built (the first always being Google Business Listing) than have 50 profiles to keep up with just for the backlinks. My point is to not forget to list your website URL in the websites, profiles, and directories that you're currently a part of. They may not be "do follow" links but they still help your website get indexed and affiliated with what you do.
I hope that long winded answer helped a little. Feel free to ask any specific questions or post your site if you'd like me to take a look at anything.