Adding bottles of formula creates a "top up trap" and decreases your supply. Your supply will automatically keep up with baby if you're only breastfeeding, he will get more efficient and feed quicker and the composition of your milk will change. Have you been expressing each time he has the night bottle?
Express - how I’ve maintained and enjoyed the benefits of both.
I'm combi feeding and it's probably 50/50 at this stage. I pump rather than breastfeed and currently pump 3-4 times a day - always before I go to bed, in the middle of the night, first thing and then try to do one in the afternoon too. So as a result he has breastmilk bottles around 11pm/3am and 6am then usually one in the afternoon but all the rest are formula. I should pump more really as I feel my supply is dropping but it's limiting!