1. Buy the items you know you're going to need often in bulk. I look for deals on wholesale lots of faucets, ceiling fans, light fixtures, etc. I buy GFCI outlets by the case of 50 and pay a little over $2 each vs $10 at Home Depot. I have a lot of rentals and add new ones constantly so that works for me, it may not be cost effective for someone with only a couple.
2. The best cost saving item I ever bought was a commercial quality power snake. If you have to have a plumber clear a drain or pay $75 to rent a power snake every time tenants clog the drains it adds up quick. I always have problems with tenants that flush their feminine items or decide to pour grease down the drain. Jeff S. is also correct about giving every tenant a plunger, it can easily save a service call from my maintenance guy.
3. I paint all of my the interiors of rentals the same color, a light beige. It makes it really easy when they move out because even if I have to repaint a wall I can usually get away without cutting in which saves me about half of my labor cost. Plus I only have to have 1 5 gallon bucket of wall paint to worry about at a time, not tons of cans to dig through. Even if the property has decent paint when I buy it we always paint it our standard color prior to renting it to save time between move outs. The same goes for exterior paint, instead of picking out colors for each properties I use 2 different paint schemes for exteriors (one beige, one green....I want an option there depending on the style of the house and what colors the neighboring houses are).
4. I never install carpet, if the place has serviceable carpet when I buy it I'll keep it. If the house is on a slab I'll tile it, if it's wood subfloors I love stick down vinyl plank flooring. You can find it for a ~$1/ft and if the tenant damages an area of it it's super cheap to take some spare planks and replace the damage.