I don't wanna!

AITA for not wanting to do visits. So my children's father and I have been handling visits outside of court. There is court ordered child support. We recently did a modification and he quit his job a week later. He cut his visitation short one week due to his GF's daughter self harming...a couple weeks latee he and his gf break up. He then tells me that his mom's house is not suitable for visitation (very nasty, never cleans and has 3 large dogs) I offer to let visits occur at my house (previously not allowed at my house but I felt this was extenuating circumstances) it was also fall break...he came and saw his kids only one time during "his" parenting time...during that visit he informed me of his ex gf and her suicidal tendencies and how she refuses to get help and that is why the daughter is suicidal and self harming...his mom chimes in with "so, I(paternal granny) am too." Dad gets back with his ex and now after a month and only seeing his kids one time despite me breaking my own rule of him not being allowed at my house and repeatedly inviting him to public outings(because I r ally didn't want him at my house) now wants them for the weekend...I'm not comfortable with it....I told him that and his only response is the kids are not in any danger... So...should we continue with our informal agreement or take a break for awhile (maybe supervised visits in public) and see where it goes(even if that eventually involves court)?