It is very very difficult to even try to advise you as you are the only one who knows him best.
You say you confronted him about the last one and only you know what he said or how he said it.
He does sound very loyal and if he says nothing went on and you want him then at least you are on the winning side and I would try to reason with him or at least find out if he truly loves you enough to drop this other woman.
It just rings a bell when my dad left my mother for someone else and he lived with me for 6 months. I used to tell my mother that as long as he was with me he was still half hers but if she forced me to send away from mine by making my life tough over it then she would lose him as he wasn't ready to return to her at that time but give him time and he would return to her if she would just stop being "I have done nothing wrong" attitude.
But no......I had to ask him to move because it was made too difficult for me. He went to the other woman. My mother could STILL have had him though if she would have listened and fought for him.
Only you know how he feels and I hope it works out the way you want it to.