Baby daddy drama

So my son's father and I have had a pretty rocky relationship. Toxic without a doubt. Ten minutes after my son was born he demanded a paternity test and treated me horribly because I said I wasn't going to force anyone to be in his life. There was constant drama between him and his mother that lead me to the decision of breaking it off and telling him to go. Eventually I got tired of him bullying me saying that everyone was calling me a whore so I got a paternity test. (Surprise surprise he was his). I tried making efforts to please his family by taking my son for visits and they treated me like I was the enemy. His mom insulted me and my mother. And then on bds birthday his pompo told me that he was going to call the state on me. The whole family was saying stuff like "I don't have any sympathy for her." Etc. I drove home crying (the drive was 2 hours.) recently I decided that I'm tired of it. I don't ever want to deal with that again. (Just so you get a good idea of what kind of mother I am, I don't drink, I don't do drugs, I don't smoke, I keep a close eye on my son, I live with my parents who are allowing me to raise him here until I decide to work, yes I have had a bit of depression and anxiety with this whole experience). Recently I've felt like the dad is pretending to be super nice. But he does backhanded things like breaking keepsakes, grabbing my arms so they bruise, and most recently sticking my toothbrush in the toilet because he knows that I'm protective of my dental care. He doesn't admit it though, it drives me nuts. I feel like he's trying to drive me insane. I've recently asked him for a week because I've been having terrible anxiety. I'm at the point of telling him to leave because I can't put up with it anymore, but I'm terrified that if I do he'll use that against me in court.... What do I do? Any advice?