I’ve inadvertently poached seats a couple of times, but happily moved when asked.
Once though I was asked to move from my window seat on a longish flight to a middle seat to allow a family to sit together. I refused, much to the annoyance of the FA. I then had a nap without a hint of guilt...