My HV explained it to me, think about when your having a conversation with someone, we never consistently keep eye contact at all times, we often look around and continue the conversation, it feels really awkward as an adult to always maintain eye contact and apparently it is the same for babies.
They get sensory overload apparently. Mine does this sometimes especially when feeding, sometimes he’s making eye contact, sometimes he’ll look away.