My ex partner/abuser wants to take our daughter away 💔

It’s a long one but I’m desperate for advice! My daughter has never spent a night away from me, she’s going to be 2 in June. She Co sleeps and still breastfeeds to fall asleep and will still wake 3+ times a night and need a feed to settle. My ex partner emotionally abusive to me and her, controlling and sexually abused me. He also has quite a temper. He’s never put our daughter to bed, never got up throughout the night and used to get very angry if he was woken up when we were together. We have been separated for 6 months and he sees our daughter for 7 hours a week. This isn’t consistent as he works away sometimes and likes to go on holidays etc and has also been drunk in the past when I’ve gone to drop her off. She comes home filthy every time and doesn’t have her nappy changed. I don’t feel like she’d be safe overnight, let alone a whole week and until she can communicate with me, I don’t feel comfortable putting her in the position. He’s on the birth certificate and there’s no court orders in place. He’s adamant he is taking her but I can’t let it happen. If something was to happen, I’d never forgive myself for letting her go 💔 I don’t want to go down the court route as the potential of overnights petrify me. What can I do 😭 I’m scared that as it comes closer to the holiday, he just won’t bring her home.