The flight number affects all sorts of things. If you change it at short notice, you need to change the already lodged flightplan, tell catering, tell the refueller, tell ground ops for bags/gate assignment, tell rostering so they can amend all the crews rosters, re-arrange arrival slots with ATC (which is sometimes impossible at short notice) etc etc etc. i don't see why it matters to the passengers so much???
I "think" the argument is the perceived unfairness where a later flight leaves before your earlier scheduled flight. Happened to me once on VA in Mel, on one of the IT check-in chaos days - I was placed on an earlier flight that ended up departing several hours later than my originally scheduled flight.