I think it may depend on the type of IRA you have. I don't know a lot about this, but did find out recently that I can't borrow from my 403b account. I can't even touch any of the money in it until I'm 59.5 or I separate from the district I teach in.
On the other hand, my husband can borrow from his 401K plan up to 50K, and we have to pay back over 5 years with 4.5 interest back into the account. We were going to use the money for a down on an apartment, but they didn't say there were specifics on what we could and could not spend the money on. It was just a cash loan.
Then we were thinking about switching over part of his 401K to a self-directed IRA, which would allow more flexibility in what we invest in. But we haven't gotten enough information on that to make it happen, or if we even can.
So, as everyone else has said, call your company and see what their rules are. We invest with Schwab and Fidelity and reps from both were happy to explain everything over the phone!