A lot of "what ifs" there. Here are a few of my own.
What if the person being displaced is disabled? Has limited use of one side of his/her body and must sit in an aisle seat on the right hand side. The family is only offering a middle seat down the back.
What if the person being displaced has a sever flying phobia, and must sit in a forward window seat? The family is offering an aisle seat half way down the plane?
What if there are any number of reasons, physical or psychological, that mean the person being displaced needs to sit in a particular seat and the family is offering anything but? You're not necessarily going to be able to observe why the person is unwilling or unable to accommodate this needy family, and it's not actually anybody's business.
These families don't only ever consist of parents with small children you know - they've been known to consist of two middle aged suits, or a husband and wife who ignore each other for the entire flight.
Sorry, you'll have to try harder to come up with a real reason why the desire of a family trumps anyone else. And don't use the entitled parent's threat of making someone babysit your child, because if you abandon your child next to me it remains abandoned. It can kick and scream all it wants, you'll be more distressed by your neglect that I will.