Keep in mind of the I/O(payments/period) and game plan on how fast you'd like to pay it down. 

If I were you keeping the property as 4 units and renting out bonus unit under the radar may be your best option especially if you plan to refi or do HELOC while occupying. 

We had a plan to build a large real estate portfolio and then sell it in five years and divide the money.

Multi-family real estate is something I am planning on doing after I graduate college and eventually full-time. 

@Matthew Posteraro That's a solid plan.

My plan is to sell my primary home in New York and purchase a house to live in South Carolina.

If I had to leave a takeaway from this post before getting too far in it would be that whatever goals are had you have, we have these goals for a reason and typically it does not go exactly as planned but even if it doesn't go as planned, we can pick up where we left off and give it a fresh start.

Many people choose NOT to have this, because the administrator will then vet every investment you make to ensure you do not invalidate your IRA.Invalidating your IRA means that you've done a prohibited transaction and the entire account is then deemed as distributed to you.If you don't want checkbook control, then I recommend a plan administrator such as Equity Trust - www.trustetc.comIf you are sure that you're aware of all the rules around prohibited transactions and disqualified people (who you can and can't do business with), then a plan administrator like Sense Financial would be a good place to start talking.  

Quote from @Daniel Tanasa: Another option is to get a small cash out refinance from the house you plan on fixing up, if is in financeable condition, and use that to fix it up, while still cash flowing.

Unfortunately, while there are other methods to acquire investment properties, the reality is you need cash and reserves to plan and set up your investment portfolio accordingly.