Not exactly true... The fee that most consumers get charged (which is usually around 3%) is most often a combination of the Visa/MC charge that the bank incurs, plus a charge the bank adds. Most companies break down the total percentage and you can see the make up of the two components.
What do you base this on? This does not happen with any of the credit cards that I currently have - which includes Mastercards and Visas issued by Westpac, HSBC and Citi. Westpac and HSBC statements just list separate transactions for "FOREIGN TRANSACTION FEE" or "OVERSEAS TRANSACTION CHARGE", without even specifying what transaction it relates to, never mind how it's made up. Citi statements say " includes International Transaction fee AUD xx.xx" next to the relevant transaction, but again they don't say anything about part of the fee coming from Visa. Can you give any examples of card providers who do what you have indicated?