Contact naps are good for both you and baby and is a perfectly normal thing to do! If he’s able to sleep on his own the rest of the day, then it sounds like you have a perfectly well adjusted baby!
Do what YOU want to do? You’ll miss the contact when babe is running round not wanting to know! Stuff the family I say!
My baby is the same! He sleeps very well at night in his bassinet but during the day 1-2 naps needs to be contact nap so we hold him otherwise he will be up crying. It’s ok!
Just lay with him or sit with him and contact nap. If baby’s happy to, just don’t!