Moving bedtime earlier

My 4.5 month old is still going to bed late (10:30-11pm) - he wakes up at 8:30-9:00am. We'd been wanting to move his bedtime earlier for a while now so that we can have some time to ourselves in the evening again. He is usually still wide awake until 10:30ish but had started to fall asleep on me at around 8-9pm a couple of weeks ago, so we felt he was finally ready to start putting to bed earlier. We stuck to our normal bedtime routine and tried to put him down at 9pm. For the first two days, he woke up after 20-30 minutes and wouldn't settle back down. The third day, he wouldn't go to sleep or settle at all so ended up just bringing him back downstairs again until 10:30 🙃. For those 3 nights, he slept terribly for the rest of the night. We've given up putting him to bed earlier for now and he's back to sleeping much better. I'll give him a few more weeks and maybe try putting him to bed earlier again after Christmas. What does everyone do when they're moving their little one's bedtimes earlier? Should we keep persisting in trying to move it forward or just keep it as it is until he's ready? I'll add that he's not the best napper during the day and will only tend to have cat naps while on a walk or sometimes after feeds or in the car