I felt the same, it has eventually lessened now though. My daughter has a lot of fun and she's always coming home with new skills that she might not have learnt at home.
Think the balance is healthy it can be tough but it is socialisation for them and crucial self care time for you just remember the guilt is what we're doing to ourselves and our babies want to see us truly happy and sometimes that is having time to recharge
I'm the same with my 20m old, she starts her visits on Friday