I gather he didn't challenge you to a rumble, then?
I've not had trouble with clothes, but one time I was travelling with an Australian friend in Wales. We'd just stepped off the train at Cardiff and into a cab late in the afternoon, and noticed a lot of barricades and police around. We asked the cabbie what was going on, and he explained that an Aust vs Wales rugby match was about to finish, and Aust had beaten Wales.
We had to walk the length of the main street of Cardiff a little while later, which was full of bars full of Taffys drowning their sorrows and spilling out on to the street. We weren't game to speak to each other, lest our accents betray us. And come to think of it, we didn't see a single person that night who was game to wear any kind of Aussie clothing in public.