AIBU is this reason enough to want to split?

Me and my partner have struggled with our relationship since the birth of our first son. There has been emotional and verbal abuse from his side and aggressive actions but never touched me. Eg slamming doors in face. After a particular bad outburst this seems to have now stopped however there is no affection (kissing, hand holding, touching) which is my issue and I can't seem to get over it. I don't really know what's stopping me. He works away for a short bursts and I find myself preferring when he's away. Most of the time when he's away we barely speak, usually 3 message exchanges a day which is him asking if the boys are ok. The energy in the house when he is back is tense and there seems to be no joy. However he is great with the boys, he plays with them endlessly and they always have the best time with him. I just feel like he absolutely hates me. The last time I spoke to him to see if he was happy he said some days he feels down and he worries about me. This is something he does every time we speak, he turns it onto me and tells me my mental health is poor. On paper we should be really happy but we just can't seem to get there.