You CAN'T avoid anybody if you gun for "rare" (read: premium) girls anyway. RNG giveth what RNG giveth.
That being said, it is the game that can be cleared with silvers and event golds, so don't worry about performance much and enjoy your time with your waifus.