Hi David,
I'm a native Lexingtonian and a buy-and-hold type investor (as of now). Let me first say that I LOVE what you're trying to do. My husband and I used to live downtown, we own apartments downtown and on UK's campus and theres such interest in people living and staying downtown. There aren't super terrible areas anywhere here, but I agree with the above-posters that some streets are unsafer than others. I would think an area you might want to expand your search is along Jefferson Street, West Sixth and streets in between. There are older shotgun style houses that investors are getting deals on before rehabbing and either flipping, or renting. This area is walkable to bars, etc. I think Manchester Street is a good bet. That area is crazy hopping right now - although there aren't many existing houses around there. NoLi is a really cool burgeoning area, but if you buy there, I wouldn't stray from Limestone itself. The streets off (East 5th, 6th, 7th and 8th), are all still pretty rough. Please don't give up on your idea - maybe try a couple of different agents who understand your vibe. It sounds like you've gotten discouraging advice about areas from some of them. Joy Moore is a great downtown go-to. About Loudoun/3rd - there's a new bus depot and lots of push to "gentrify" this area, so you actually might be a bit ahead of the curve here. There are lots of new sidewalks, etc, but the closer you get to Loudon, the less walkable to downtown it becomes. I might also expand my search behind Rupp Arena, the "Newtown Pike extension," which is near Alltech. Good luck - Lex is lucky to have someone like you interested in doing this!