OK, none else has said it so I will.
1) You deboarded the aircraft that was in the process of boarding holding up pax who would have been embarking (I have seen people walking back up the aisle during boarding and it is incredibly disruptive as pax have to move themselves, and their luggage). Presuming you were on a 738 it's a single aisle aircraft and you and your wife are coming all the way back from row 15.
2) As such, you did hold up other pax, the crew and the plane
3) There is no way a ground operator, charged with the responsibility of ensuring on time performance, and who is measured accordingly, would tell you to go to the lounge yourself.
4) An entire aircraft of pax, the aircraft and the crew were delayed by your actions. Every minute counts to an airline. Why do you think they have such strict cut off rules? The pilot would require an explanation and it's possible the paperwork could not be completed, and the doors certainly not close while you were running your errand.
5) Did you leave your toddlers unattended in row 15 while you were running through the airport?
Excuse the rant but when one pax put themselves above all the others, and inconveniences everyone, then decides to complain about a CSM, just doesn't sit right with me.