Boyfriend with 2 kids

Hello! I have been in a relationship with my boyfriend for about 3 months. I have a 2.5yo daughter and he has a 7yo boy and 10yo girl…so we both are busy with the kiddos 😜. I try to see him as much as I can even if it is just for an hour. Last night I went to his sons soccer practice that he was coaching and then met up with his family for dinner after. Upon leaving I said “ see you later” I didn’t mean to say that as he didn’t want his kids knowing. The plan was for him to come over and spend the night once the kids were asleep. He lives on his parents property so they are in the house with the children. His children heard me say that and I immediately got a text of my bf saying they heard it and that he can’t come over anymore as they want him to stay. I understand his children come first but also I don’t get much 1/1 time with him. Our schedules don’t always line up being we both are co parenting. It has been 2 weeks since I have had 1/1 time with him. In my opinion I see it as harmless if his children know he comes over when they are sleeping. He has the luxury of having family here who can help out with the kids occasionally. Better then lying to the kids or sneaking out! Anyone have any advice/ opinions. This is my first time dating someone with children. Hard to find that balance.