He’s your baby. It is your right to protect him. So don’t be afraid of how your uncle will react. Simply send a text even explaining you will no longer bring your son around them or just stop bringing him with no explanation. My son is also 3 and he’s a sponge. So it’s better he isn’t around that “grumpiness”.
Don’t make him stay there if he doesn’t want to. What’s the point? That’s a huge red flag. There’s a reason he doesn’t like staying there. If you’re afraid of your uncle being mean to you when you break the news that you don’t want to force your kid to stay then imagine how afraid your kid is. Your babies feelings are more important than your uncles.