New landlord question

[Ryan King]

Hey all. First time poster here. Thank you to all who read this. 

I bought a BRRR in January. One of the two units was rented, month to month. I had the tenants sign a month to month lease with me, changing only the late fee after the 5th to $50 from $25. When I went to meet the tenants on the 5th for the first time and collect rent, they told me they didn't have it. So I accepted it on the 15th with no late fee. They were late one other time and I did not charge the fee. These tenants were relatively quiet and clean. No issues EXCEPT when I was working on the downstairs unit with adequate notice that I would be doing so, they'd come down and ask us to stop because they were napping. This is an issue because I work full time and my husband is frequently out of the country- so when we have the time we need to work. This was always early afternoon (1p and 3p.)

I opted to not renew their lease in April and informed them this would be their last month of tenancy on April 2nd. Asked them to be out NLT May 2nd. They were mad. Told me they wanted all of April rent back, etc. I provided them with a link to the landlord tenant act and a copy of their lease. 

They moved out early and I am working to get the unit rented. I have the units professionally cleaned after move out at the expense of the tenant per the lease. I provided documentation of this and their refund today. They also made 8 additional copies of keys. I asked how much they paid and they didn't remember, so I gave them $2 per key back. They are now saying they plan to take me to court for the remainder of their deposit and the rent from the date they moved out. 

Do I have anything to worry about in this situation? Thank you.