You are right, for quite a long time the ladymen have been going to the female toilet and it has never been a problem, until now. The reason why is the real issue with gender and the toilets, is not about the toilets, the real issue with gay marriage is not about the marriage either.
The socialist agenda on this and many other issues is really about the legalisation of these social behaviours (for lack of a better term) and what will come about after that. Despite the fact that modern society is tolerant of diverse social behaviours (note that tolerance means "to put up with without liking it") the rainbow people do not have 100% acceptance. Legalising their lifestyle is another step towards forcing everyone by law to accept their way.
Many people are a champion for this cause because they think that anyone who doesn't wave the rainbow flag now is a religious bigot, but what they are voting for is taking peoples freedoms and liberties away.