I guess it becomes a downside for those who do not have any lounge access. I suppose the argument is then that such people likely don't travel enough that paying for a meal on board, one in advance at the airport or paying a higher fare to get a meal included, is of little consequence to them overall (i.e. "no big deal", you pay it once or twice a year, that's it).
If you "travelled often" but had no access to lounge and had to pay for meals every time because it wasn't supplied on board, that could be something else. The only people I can think that fall into this are those who maybe travel at least once a month across the Tasman in the cheapest fare available. Some of them might just write off a meal expense (or even a higher fare) on an expense account - no big deal. Some might just call it the cost of doing business (or travelling).
Some people, irrespective of status held, have argued that they do not arrive at the airport for such flights early enough to enjoy a proper meal at the lounge or even from a food facility at the airport. They "rely" on food (decent food?) being available on board to tide them over in their very busy schedule. Thus, they could be key opponents of the fare structure.
It could just also be a whole perception thing. Airlines are sometimes in a bind on this one. They are "responding" to consumer surveys which, including people like you, indicate features they are happy to forgo in order to pay less on fares / keep fares lower as long as possible / maintain schedules. But from an outside view, when features are taken away like that, it looks like a cheaper low cost structure. People who don't take the time to do the sums in detail may make a sweeping negative conclusion. That may not mean much overall, but it could tip someone to the competition, too.