I would take the extra day with them and try to make the absolute most out of your time. Get to know them better ...find common ground. Its not too late to have a better relationship. If you have children, it's so good for them to have solid relationships with their extended family. Good luck!
I get it. It's hard when you're not super close, and I understand how you want to get things back in order and have your own space. I get a bit frustrated when family visit...or I visit them for similar reasons, but I try to remember, its a couple of times a year and it's only a few days at a time and those few days are doable. Plus, my son loves it. Maybe they could give you an hour alone time to have a nap or to do something just for you? That way you get a bit of time to reset and get you through the extra day. I would just go with it to avoid an argument and try to make the most of the time, but ultimately, only you know if you can handle that extra day.
I agree with all the other comments - especially if you don't see them thay often. Sounds like your dad is trying to make an effort too. Maybe once Sunday comes around you can say something like you need a couple hours to do some life admin before the start of the week (batch cooking, cleaning etc.) So would they mind if you had a few hours to yourself and you can have a nice dinner together in the evening to make up for it?
I think given you only see them a few times per year it would be a bit miserable to tell them no. It is your choice though