Everyone has good points on this in my opinion. Diversification will always have credibility and be the "norm" in traditional investment theory. This is a basic and fundamental concept, such as risk vs reward.
With that said, you will find many very successful investors that were not afraid to put all their eggs in one basket. And those which exceed the norm, do not do "normal" things. Of course, no one could recommend that you go out and do that. The common theme that you find with these investors that are successful long-term (not flash in the pan types), is that they educated themselves to the point that they became experts in their field, surrounded themselves with the right people, and they are likely still diversified in that broad assets class.
The truth is, most of us should not be on completely on one side or the other of this diversification scale, but rather somewhere in between. Exactly where we properly lie on that scale is based on our individual risk tolerance, our own expertise and competencies, drive, and our time devotion. Everyone's answer will be different. Ford or Chevy? Republican or Democrat?
My personal approach, is to increase my real estate holdings proportionately with my knowledge and expertise in the field. The more time I devote in educating myself in the particular investment, the less risk I am taking, and the more comfortable I am in moving money from the traditional assets to real estate.
For Self Directed IRAs; I would suggest researching before jumping in, but they can be a useful tax tool for sure, depending on your strategy. Some things to consider that I have learned in my experience with these are; look into the fee structure (typically based on the value of the portfolio), and tax advantages vs disadvantages as mentioned by others, the credibility and strength of the custodian (look up American Greed for a fun story), and potential tax return filings each year depending on your structure and investments (ie UBIT/990 and 568/CA LLC).
Regarding the Roth IRA accounts mentioned; for those who look to retire before 59 1/2 years old, look into the "Substantially Equal Periodic Payments" exception. I hope you and I are in the position to use this election soon.
@Michael Swan that's awesome! Thank you for sharing your story-inspiring...