Then you would have this BS argument involving the FA every time when someone has a bag in your slot. The argument would proceed and if you're lucky you'll get your spot and the offender might get whatever is coming to them (or they may get lucky as the FAs check the manifest for a slot that won't be taken because there is no pax allocated to said seat). Either way, a waste of time for all involved.
Then you'd have some that would shrug and just place their bag in any empty slot rather than disturb the errant one in their slot. If that causes a chain reaction, then we have another huge waste of time as we track down who caused the whole mess first.
I guess we're back to air travel would be so much easier if everyone could be reasonable and be trusted to do the right thing.