I experienced this too and what i learned from it is that your baby is comfortable around you. Babies dont know the other person so they can’t trust someone else holding them so they’re more quiet yet alert because they dont know whats going to happen. Your baby can trust you which is why he’s more active with you. It used to get me down until i realize that my son feels safe around me and that its okay if he gets unsettled when hes with me. We are both learning. He also had a period where all he did was cry as a newborn even though i did everything. I did some research and found out newborns will go through a phase of just crying quite a lot for a week or 2 because they dont know how to communicate/regulate themselves yet so they just cry just because. That calmed me down a lot too. I google everything since im a first time mom and its helped me and i feel more connected to my son rather than feeling lost and incompetent and comparing myself to others.
Agree with the above comments and you may want to factor in other ways to soothe. Singing, humming, swaying etc. I figured out my daughter likes when I sing in a higher voice than normal for her lullabies. Learn a lullaby you won't get sick of, I did Skye boat song. Your baby loves you and you are their safe haven. It's hard to hear them cry but you'll find ways to soothe him and they'll work like magic.
Do you nurse?
My baby used to do this too but then the doctor said it's because they expect to nurse whenever you hold them and cry when it doesn't happen. I switched positions, started holding her differently than when she would nurse and started humming, talking and rocking when she would cry and eventually it got better and she calms down in my arms now too.
They can sense when you feel nervous so I’d say try to be calm when holding him! You’re doing amazing ❤️ sending you hugs