The advice I give is the advice of an experienced owner-operator in Brooklyn. Understand that maintaining the property free of vermin is your responsibility. That said follow these steps and you will be fine. Ignore them at your peril.
1. Hire a licensed pest control company. Don't hire one of the big guys, hire someone that does the work himself and owns the company. I can recommend one or 2 offline if you wish. They are the boots on the ground and tend be be better than a big company that just employs some schmuck who doesn't care.
2. Walk the property with them. Have them show you the possible points of entry. With mice it' ALL about closing points of entry. If the problem is as bad as you say, you have multiple points of entry and breeding. Close all the open areas and fix things that need to be fixed (think holes around radiator pipes, behind the stove, door sweeps). Fix them ASAP. This just needs a handy man.
3. Pest control really only works when the whole property is treated. Make sure you secure access to the whole building. Make sure trash areas are kept clean. Make sure no trash if left in the hallways. Explain to the tenants they are NOT to store garbage under the kitchen sink.
4. Do monthly service. As winter approaches the problem will get worse, not better.
5. If you ignore the above, the tenant can call HPD which will, I promise you, be much much much worse than paying $100 month for an exterminator. You will be on the hook for the exterminator, whatever repairs HPD will require you to make, AND the tenant can stop paying rent claiming the premises are not livable. And you cannot recoup lost rent. AND you cannot evict them for non payment. And you will not be able to do a single thing. I don't even know anything about FEPS but if it's a govt program it will make everything worse.
6. Welcome to being a NYC landlord and I promise you the advice above is worth it's weight in gold.
7. Oh, and where there are mice, roaches are not far behind. Make sure you get the property treated for roaches.
8. Also, it might be worth it to let the tenant have the cat. It is a genuinely good deterrent of mice and you don't need to pay of it ;)