My theory is a lot of the time 4B and 4E are filled manually. I have come to that conclusion from talking with people who end up in these seats. The next gen check in system seems to do a good job of keeping them free. However check in and service desk staff rightly or wrongly seem to be quite happy to fill them even when there are window or aisle seats available near the front. This mainly seems to be passengers on international connections, passengers on connections that orginated from small airports without self check in, passengers who missed a flight/connection or asked to be moved on to an earlier flight.
You are probably right, but in saying that I have been on lots of flights where rows 4-7 or so have the middle seat blocked and when going to use the rear toilets I see that in the last 10 or so rows the middle seat is mostly occupied.