There's nothing wrong with it. It's okay to drink less. The protein content of milk is very low,100g is a few grams, mainly with high ca+ content and then water, which is a little fat, so milk has no effect, just don't go to the toilet or upset your stomach after drinking too much. Protein can participate in the synthesis, but it is not dominant.
Does not affect weight loss, as long as it does not conflict with weight loss.
There are individual differences for reference.