I must admit to not reading all of the preceding pages of responses however the few times I've had someone sitting in my seat, I stop to face them, take my time to look at the boarding pass.....look at the seat number and then back at the boarding pass. Provided I am in the right, I then say in a normal volume (ie not a whisper), "excuse me, you are in my seat". For me that has always been followed by an apology and the person shifting (genuine mistake, I presume). If there were to be further debate, I would ask something like "so where is your real seat"......and if necessary, "I'm sorry but I chose this seat some weeks/months ago because this is the seat I wanted and your seat is not what I want".
There is nothing wrong with standing by your guns. I have experienced the uncomfortable feeling of others tut-tutting at me but let's face it, you're unlikely to ever need their uninformed approval anyway and it is possible that the by-standers are really taking your side but your embarrassment may actually be blinkering your perception a bit.
By all means, consider the offer, but if it's to your detriment, let them know and suggest they move back rather than expecting you to do so.