Part of the issue is ABF staff at the border do not know their own processes as you've experienced. The procedure for dual nationals is well documented on the ABF website publicly.
Dual national passports should be linked in the ABF APP (advance passenger processing) system. These links last for the life of either passport so it is a process that only needs to be done once each time a passport is renewed.
This linking process can be done by a travel agent who is familiar with it or airline staff at airport check in.
As
@Princess Fiona has already mentioned, once the passports are linked, you use the foreign passport at BOTH check in as well as the outbound smartgates.
For example, if you are a dual GBR/AUS national and entering the USA with your GBR passport as your ESTA/Global Entry is on the GBR passport, you would check in on your GBR passport AND also use the GBR passport to exit Australia at the outbound smartgates.
This issue will become more prevalent and more important next year when the UK and EU travel authority systems come online for Aussies, so it'll apply to travel to Canada, USA, UK and the EU. Given the sheer number of dual GBR/AUS nationals in Australia, it will become quite a common procedure in future.
This is actually not correct procedure. See above.
No. There is a well established procedure for this that airlines should be/are aware of. The process involves obtaining a manual authorisation to uplift the pax from ABF.