You don't need a portfolio loan to build a portfolio of rental properties. I think the term "portfolio lender" is more clear. You can own 10 properties with mortgages from 10 different lenders, one (or more) of which is a portfolio lender.
From my understanding, only a conforming mortgage can be packaged and sold off. A confirming loan is one that fits Fannie's and Freddie's standards. I believe these are the only mortgages the big national banks will do. A smaller community bank often holds the mortgages they write and do not package and sell them. Because they don't intend to sell these mortgages, they don't have to meet the standards. For instance, a portfolio lender may write a mortgage for someone with a 700 credit score where a larger bank may not.
In short, portfolio lenders can be more flexible and may "bend the rules" to make a loan they consider a good investment. I personally feel building a relationship with a local portfolio lender is a good move, but not necessary to obtaining a portfolio of rental properties.