I've been analyzing deals using the various calculators and have some of my own cash to get started.I'm wondering if not using a buyer's agents sweetens my offer to a seller, who then only pays one agent, not two.

You will really have to sweeten this deal for them, higher interest, higher purchase price or both.

I spoke with my wife about it and she said we should hear him out, because (funny enough) we had actually been discussing possibly downsizing in the future because we might be going down to my income only.A few days later he texted me again with an offer - they would buy the house back from us for what we paid ($355k), provide $50k to sweeten the deal, and pay $5k toward our moving expenses.

Both of you should be thinking, "how can I sweeten the deal for the other so that we can do many more deals together?"

Typically there is an additional sweetener called 'preferred return' that you would offer in these situations.Returning the principal is pretty much a given.

Another option is to offer a private lender equity in the deal to 'sweeten' their share. 50% of a deal is better than 100% of no deal.Did your contractors give you quotes?

You can mess with the numbers all you want (IE argue that vacancy won't occur or you're self managing,) but that's just trying to sweeten a bad deal.  

He gave me a discount every time I went in which seemed nice on the surface but I soon realized that he was probably trying to sweeten me up for something later down the road.

You can even sweeten the offer by paying I/O payments, during that stretch, to reassure them.

Appreciation is where you make the real money, and cashflow is the sweetener.