IIRC, the fee was changed to one-way only long before the loan was paid off and was doubled to allow for the change. Saved money by reducing the number of toll collectors and sped up traffic flow in one direction. At that time the toll was costing more to collect than the amount raised.
I can remember people discussing crossing the bridge in the free direction and returning via the Gladesville Bridge to avoid the toll.