Nothing I could do (believe me - I tried just about everything mainstream), could turn him on. I've never been involved with a man who had erectile dysfunction, but I did date a man last summer who indicated he had little interest in sex.
I had my own sexual desires, needs, and did not want ot live the rest of my life in a non-sexual relationship, and could not ever cheat on him, so I filed for divorce. I was with my ex husband for over 12 years and the sex was great, but shortly after he turned 40 his sex drive took a nose dive and never recovered. In fact, he said he could take it or leave it, and didn't miss it at all.
When he stated you were stressing him out, it was kinda like he has admitted to falling out of love to himself, and this is his way of telling you, by blaming you for his lack of sexual desire. He has realized he has lost his sexual desires, and understands the impact it has on you, and feels it's best for him to 'let you go' rather than stand by and watch the both of you fight a losing battle. At that particular time in my life, I didn't either, so we were an ideal fit.
We found a basis for a relationship that did not involved much (if any) physical intimacy.
~OP~ I wouldn't take it personally, he just wasn't the one for you. If a relationship is causing more grief than happiness, it's time to move on.