I know multiple people that live an amazing life purely as passive investors. They had two things in their favor when they started; 1) - they started with roughly 300-400k and spread it over multiple deals and 2) - they got started in the early stages of the recovery (2012-2013).
It is possible to build a nice passive income but as shown above it usually takes some decent capital and the right time in the market to really get it going.
The folks I referred to are making 200-300k a year passively and have grown their net worth to between 2mm and 4mm and travel all the time. This happened because many of the deals they invested in doubled, tripled or even quadrupled and have also done many re-fis or supplemental loans along the way that grew their portfolio exponentially. This good fortune also had some bad deals along the way but has worked out well for them.
As mentioned by others, there are definitely pros and cons but if you have the capital and patience you can build quite the passive portfolio.
Caveat - obviously, these folks go involved at a fantastic time that may not exist currently and they started with a fair amount of capital but you can still start now and end up with a nice little nest egg that pays for a pretty good life passively.