In most jurisdictions a landlord must have receipts and have work performed and bills submitted by licenced contracts in order to justify withholding security deposit.

And, the housing authority can withhold funds if you haven't passed an inspection, etc.Basically - if you don't have to accept Section 8, then don't.  

As I recall you have to send them why you are withholding the deposit with specified damages within 45 days.

However, a large number of holes in the walls or ceiling that require filling with plaster, or that otherwise require patching and repainting, could justify withholding the cost of repainting from the tenant's security deposit.

I once had a former PM withhold part of a tenants deposit when I switched PM's.

There are often only very specific circumstances where you can keep a deposit, and sometimes there are triple damages to pay if you withhold it illegally.

If you withhold some of their money and they sue you, and the judge says you have to pay them, you pay them.Really, when would the issue ever come up in real life?  

This is actually why most tenants who move away never sue their landlords for illegally withholding their security deposits - because they have to sue where the contract took place.If you get a judgment first, a collector will take a smaller percentage.  

Withholding rent is one of the only recourses a tenant has against an unresponsive landlord outside the court system.

I think it would be good to send him a text similar to the following to document your concerns over him withholding the security deposit.