Financing a private island problem

[Gerard G.]

I'm looking for options to refinance one of my investment properties that has unusual circumstances.  I'll give a brief rundown and any advise would be appreciated...

1) Bought a private island 2.5 years ago with the owner offering seller financing.  Closed on the property with a land contract.  

2) Put 15% down and amortized the loan over 7 years so a lot of equity could be built by the time the balloon would be due in 3 years.

3) Contacted my normal mortgage broker through Guaranteed Rate, said should be smooth sailing.

4) Made improvements on the property and the appraisal came in higher than my purchase price.

5) I now currently have over 50% equity in the property.

6) I have excellent credit and plenty of room with my debt to income ratio.

but...

My broker has exhausted every option to get this mortgage refinanced over the past 3 months of shopping my package around.  Because the island has 3 "accessory units" on it (in addition to the main lodge, there are 2 cabins and a livable boathouse) every Fannie Mae and Freddie Mac lender's underwriters denied the loan because of an underwriting guideline only allowing 1 accessory unit.  Why is this even a rule!?!?

My broker's private bank and private lender options also denied it for being too unique of a property (it being an island) and had a concern about not having year-round emergency vehicle access (it's in Northern WI so we winterize it Oct-April).  My broker is at a loss, as am I, and said he has exhausted all his options that he is allowed to sell to.

Does anyone have any good advice on any lenders or companies willing to refinance a unique property like this?  I only owe roughly $350,000.  I am at a loss on why this was turned down when I thought I had all my ducks lined up!  Thanks in advance!