Am I wrong for refusing contact?

My daughter's dad is a narcissistic a-hole. We've had a custody case open for a year now because he started making threats about taking her to live with him full time after I got engaged. He started constantly accusing me of dumb stuff like I was being neglectful to my child so I served him with papers. For the record, I was never against the idea of her spending the school year with him IF it truly seemed to be best for her. But she is thriving in every way with me and when she's with him he barely keeps her hair combed, dresses her in too small clothes, don't make sure she brushes her teeth, and was driving her around without a car seat since she was 3. After all these issues and arguments over the years I am no longer open to him having primary custody. He periodically texts me long messages complaining about the fact that I live out of state. It's a pointless conversation because I have no plans to move at this time. It's not impossible in the future but not in the plans right now. And I'm not willingly giving him full custody. He keeps claiming he's ready to sign the custody agreement but here we are a year later and $10,000 in lawyer fees and still nothing. I feel like he is just trying to stall and hope I run out of money or give up. I'm not going to. At this point I barely respond to his pointless texts and refuse to answer when he calls. I told him yesterday if he doesn't plan to sign in the next few weeks that I'm proceeding with mediation/court because this is ridiculous and I refuse to keep dragging it out. Now he's mad because I didn't answer the phone when he called twice yesterday saying I'm being disrespectful and not wanting to communicate cordially about our child or case. On one hand I'm like am I'm wrong to refuse to communicate and come to common ground for our daughter's sake or is it understandable to just be over all the pointless conversations and disrespect?? His lawyer excused herself from his case last week because he's a disrespectful asshole including to her but I just feel like anything that needs to be said about the case can be said to my lawyer. I'm so over it. Keep in mind he's gotten out of a whole year of paying child support by dragging his feet as well.