When to remove financing contingency? Upon LOI or commitment?

[Sean Murphy]

I'm working on a 5+ unit purchase and have an LOI from a lender but a financing commitment will take a few more weeks beyond deadlines that have already passed. Is it common to remove a financing contingency based on a lender's LOI or wait until until a full commitment? I'm extremely reluctant to put my deposit at risk by removing all contingencies when any number of issues could prevent a lender from financing through no fault of my own.