@Aiden Huynh, hi and welcome to BP!
Anything is possible, but without knowing anything about Canadian real estate law I'm not sure how helpful this will be. I will assume some similarities with USA law, since we have a common law heritage from British Empire days.
First, you need to find out how much is owed on the property. If it is free and clear, you could possibly use owner financing, though if you have multiple extended family members that could get complicated.
Second, you want to decide the best approach to purchase the interest extended family owns. It would likely be simpler and "cleaner" to buy out their interest. You could raise money from private investors or choose to go the bank route. Family complicates things: there would be issues with who owns what percentage of interest, what to do if one of them wants to cash out totally before the others, what would happen to their interest if someone passes away, etc.
Third, what is the current market value of the house. "A few years ago" is not today. Get that value so you can decide how to fairly price their interest. In the US, we have Brokers provide a "brokers' opinion of value" that may be sufficient for what you want. Usually only costs about $100 - $200 for a single family residential house vs. a full $500 appraisal. I don't trust values from the web like Zillow or the county assessor.
BRRRR strategies implies increasing value so you can refinance and cash out. What kind of value add are you proposing and how will you fund it?
Also, can you clarify the situation? It sounds like your mother only owns 50% of the house today, but she wants you and your brother the own the full 100%....right? I don't know how inheritance works in Canada, but in the US if she gifted you her ownership interest or sells it to you at below fair market value, she would owe gift tax on the difference and also the tax basis would be recalculated today. In the USA, if you inherit the property, then you receive it on a stepped up basis and pay no tax if the property is sold at the appraised fair market value within 6 months of receiving ownership.
Consult a local Canadian tax pro so you understand all the tax impacts of any arrangement that is more complicated than a simple sale.