Just like short trips being of concern to some uber drivers, some uber drivers also discriminate/knock back low passenger ratings too.
Doing what you suggest on the odd occasion may not have much of an impact, but if you're in the situation where it's a typical route, and you do get lots of inquiries and approach it that way often, your rating will take a hit. Which may make it harder/longer to get a pickup, and worst case results in 're-education/lock-outs' by uber: Uber riders with low ratings to be banned under new rules
I'm not defending the call up practice - just pointing out that the rider rating does have significance.
How many drivers actually care about the passenger rating? I don’t. And in any event, the odd low rating from an irritated scumbag will make no difference to anyone who users the service with any regularity because almost all the other trips will be 5 stars.