Hello @Craig Fitzpatrick II,
First of all, congrats on your first Duplex. I think taking advantage of the FHA program for that was very wise of you.
To answer your direct question about personal credit -by putting a home mortgage on your credit almost always benefits your score in the long run. Obviously you have to prove that you can responsibly handle that amount of debt by never missing or being late on payments, but if you are responsibly paying off that debt than it will increase your score.
For your next purchase, you will not be able to use the FHA program again unless you refinance your current duplex into a different loan program as you cannot have more than 1 FHA loan at a time.
However, using Fannie/Freddie as a conventional loan is completely doable. You will just need a minimum of 5% down, and of course make sure all your DTI ratios are qualifiable. Good news is, there are multiple different ways to possibly include the income you are making from the duplex to offset that mortgage cost for your next purchase.
Let me know if you have more questions or would like to talk through the finer points of things as you look for your next one!