Help / advice

So me and my baby dad have had a on and off relationship since both I got pregnant. He was never happy that I got pregnant and has told me many times in the past that he kind of resents me for keeping it and having my son. My son is now six months old and he’s never really been a big part of his life, but we recently went on a mini holiday with me,my son, my son‘s dad and his daughter( from a previous relationship) During this holiday some things happened that has not made me feel safe being near him and made me worried for my child safety around him so lots of thinking about it and speaking to some close family on both sides. I have decided to stop all contact. For our safety this all happened over a month ago and he did not care when I spoke to him about it.( he has no parental responsibility) as of yesterday I had a message from his mum asking to make plans for my son to go and stay with them for the night in the past this has happened and I never had any problems but after that holiday I do not feel safe sending my son to his grandparents house which his dad also lives in. I am unsure how much he’s a family knows about the situation. Anyone have any advice on how I go about letting them keep a relationship with my son without telling them any information or possibly putting my child in a situation where he’s not safe? I have no problem with his grandparents and would love them to have a relationship with him. Someone, please give me some advice or help me 😂