I've been in a similar situation.. I stopped contact and was pressured by him and his family to reinstate it. I spoke to social services and was told it's my responsibility to keep little one safe and they can't do anything 🙃 so for now he only sees little one under supervision
I would stop contact and think about seeking advice from children’s services if you want some professional weight behind your decision?