I am a contrarian to what you have said. First, real estate is not passive. Rentals are a business and it comes with work like any business. A tenant of mine had a sewer break yesterday and I had to be there. I have one vacancy and it has to be filled. I have a pile of bills on my desk and I have to pay them.
For those who hire property managers, at best you have to be on top of them. More often, they do not take care of properties as you would.
The returns for quality real estate is actually pretty slim. You have to have quite a few doors to make a decent living. And you have to have capital to get to that level. Properties also pay themselves down slowly. If you have a 30 year loan or even 15 year loan, it just takes time. I find that a few properties of mine every year have a major capital need. That wipes out the cash flow for awhile. Roofs are expensive.
Here is a positive: real estate may be the best retirement. Long time landlords that I have met, who have patiently paid down their buildings are sitting a lot of equity. They have owned the properties 20, 30 years. But it took time. At age 30, most of us are towards the beginning of our landlord careers. Astute investors can trim down their portfolios in time to keep low maintenance high yield properties. And have a semi-early retirement where you work 15 hours a week. Now that is realistic.