@Jamarique Wynn it’s hard to answer your questions without knowing the specifics of your contract. Are you making milestone payments as work is completed, or did you pay him in full up front, or something in the middle?
Assuming that your contract has terms that cover how the contractor could breach the contract (eg not completing work items in a documented prescribed order) then you would have grounds to fire him. Depending on how you structured payments in the contract you may owe him funds, he may owe you funds or even possibly in the middle. This area is almost always open to some level of subjectivity.
As for asking for a refund it woudl also come down to what your contract says and also if this is refund for work performed or for advances. Getting a refund for in-scope work, that was performed out of prescribed order, but work you still paid for - would probably be difficult.
Lastly, taking a contractor to court for unfinished work seems like a waste of time (again unless you mean you paid him in advance and want some of the funds returned post firing). Court will take months. Court will have costs.
Changing contractors in the middle of a job will almost always cost you time and money, so unless your contractor is incompetent or a cheat, it’s almost always better to try and see if you can find a way to complete the project with the same contractor.