Underweight

My son isn't inherently underweight but his weight has fluctuated up and down and now isn't going up and the Dr is concerned. We had an appointment with his PA this is what she sent me after the appointment (with some words paraphrased for privacy) "I have discussed your son's weight trend with his doctor and she agrees to increase solid offerings to 3-4 times per day. You could also begin supplementing with formula (1-2 bottles per day in addition to breastfeed), especially if concerns with your milk supply. However if increasing his solids going well, ok to wait until his weight check with his doctor. If there are still concerns with his weight gain at that time, scheduled formula supplementation will likely be recommended at that time." For the record I dont have supply issues in regards to breast milk. This was 2 weeks ago, his appointment to follow up is 2 weeks from now. Increasing solids is proving difficult as he has entire days where he refuses to eat more than maybe 4oz total then other days where he eats 3 perfect meals but when he has 1 or 2 low days his weight goes right back down. He can't maintain anything he's gaining if he refuses to eat. We give him a variety and on the days he refuses it's things he normally enjoys or he will eat very little then refuse. With food he doesn't like he makes a face and spits it right out so ik that's not the problem. We haven't tried the formula yet I was hoping to see improvement by now. He doesn't like bottles to begin with. He will only take a bottle of BM if I'm not around and last several times we gave him formula (sporadically over the last several months) he was only interested in chewing on the nipple and spitting/spilling the formula all over himself and others. I'm so anxious at this point what do you do if your baby refuses food?