Yes. There are lots of downsides to seller financing. The terms will be worse than a bank loan: shorter terms (higher payments = less cash flow), higher interest, usually higher down payment, etc. Most sellers do not want to do seller financing because they take the risk of non-payment and having to foreclose, which is a lot of work most sellers don't want to deal with, so there are considerably less of them.
And most seller financing deals in my market are usually just plain bad deals, because 99% of buyers have conventional financing or cash, which is WAY more desirable to a seller. So, at least in my market, sellers that offer seller financing only do so because the property has major issues keeping it from being financed in the first place, or they are can't get any other offers for whatever reason (undesirable area, etc).
There's a lot of talk on BP claiming seller financing is a great way to get into investing. I disagree. It might be your only option if you don't have the income, credit, etc to qualify for a conventional loan, but that doesn't mean its a good option. In my opinion, it RARELY is a good option.