I think it's worth evolving your high level 4*s first, they only need to reach 50 and a few of them are close. It's very easy to raise them back up to 20+ so it will be a big boost in power if you have max affection.