Am I too controlling?

My baby daddy has a criminal record and tends to hang around a bad crowd. He says he doesn’t anymore but I am so used to him lying that idk what to think anymore. He lost his job a couple months ago for stealing from the company, got a new job and lost it for fighting at work, got a new job and was recently suspended for potentially stealing a weapon from someone’s car (no proof yet) and now works back at a fast food restaurant again. He has stopped helping pay for my son (we have never been to court, he paid support voluntarily). But now he can’t keep down a job, says he had no money (his cars going to be taken by the bank soon) but is getting new tattoos, going to clubs every night, going to six flags with gang members so idk if he’s lying. He’s hotboxing his car and then expecting to pick up our son (4, who has asthma). He never shows up on time to get out son for their time together. So I finally set some boundaries and said if you can’t show up on time for him, in a clean car then you’re not picking him up. He showed up an hour late a couple days ago (with no communication he would be late) and I refused to let my son leave. He showed up at my door knocking and ringing the doorbell and yelling for 2 hours. Idk if I am be too controlling but I just feel that if I don’t keep my word with my set boundaries that he will keep pushing them and won’t ever respect them. What do y’all think?