I don't think there's an issue with the smallest people, was in the Car lounge the other day with a small family in there. Kids were doing a little running around but they were cool, enjoying an adventure etc. But they may have been with a parent full member. Also may depend on the exact terms of the card. Not sure if they're all the same. I'd look on my c/card website first and wants second, even though Qantas will have final say. At least gives you some ammunition if needed, given the prevalence of free passes out these days.
If they said no I think it would be maybe more to do with the day - a crowded lounge due to flight delays etc etc. And some lounges like Sydney and now Mlb always seem to be busy these days. Remembering that a REALLY crowded lounge can even see them turning members away.