Refusing dinner now it's dark in the evening?!

This is a weird one. My son has always been an ok eater. Not terrible, not amazing. Just ok. He eats pretty healthy, he likes almost all fruit, and there are a few veg like broccoli, peas and sweet potato that he'll eat happily on their own if he has a bit of ketchup or butter with it. Other vegetables do have to be somewhat hidden in meals, but it's no biggie, no more so than most of the toddlers we know. The problem we are facing now, is that when it gets to about half 4, he starts going on and on about how it's dark outside, and asking if he's going to bed. He doesn't seem tired when he does this, this kid is a maniac all day, but he has always loved his sleep and his bed. It's like he thinks "yep, it's dark, I'm meant to be going to bed now". That means that dinner is a nightmare. He says it's "boring", he doesn't want it, doesn't even want his dessert of grapes or yoghurt. He is especially tricky when his dad and I are both home. We aren't having anywhere near as much issue when it's just one of us, we are only all home together one day a week, the rest of the time one of us is working a 12 hour shift - so meal times when we are both there are infrequent. I always eat with him when it's just the two of us, dad doesn't always, makes separate meals and eats at different times (he eats a lot of stuff my son and I don't like), but he will sit with him and chat and have a cup of coffee. When we are both home it's like he acts up as much as possible with meal times..It winds his dad up more than it does me, and I think it's becoming a "thing" now, where he will specifically play up so dad shouts, then I get annoyed at dad because I can't stand the stress and the shouting, so he successfully breaks our united front! That seems to be a main goal at the moment. I feel awful because tonight he just totally refused, dad said "ok well you will just go to bed without dinner or pudding then", and he just happily said "ok dad, come on let's go!". That stumped us. So he's gone to bed with no dinner. Is that ok?? He's not a huge kid and him not eating well stresses me out! He eats poorly at nursery as well from what I see on the app, eats all the puddings and fruit of course, just picks at the main courses.