Does paying off a mortgage early affect future loans?

[Matt Wan]

From my understanding, mortgages are considered riskier investments than most other loans because the borrowers can repay early with little to no extra cost. If I hold a 6% bond and the rate on new issues drops to 3%, then my bonds are worth more because they pay more than the new issues. If I hold a 6% mortgage and the rate for new loans drops to 3%, my borrower will probably just refinance (thus the invention of the CMO). Even without changing interest rates, the duration of a mortgage is never as certain as that of other bonds. 

In light of this, would lenders see me as a higher risk if I have a history of paying off mortgages early?