Depends on if you want to gain all the content in the game or not. I'm at 30+ girls 100% right now and have started on 2* girls as I have no more 3*+ with a certain gift type to lvl affection with.

It's worth getting 100% for your 4*+ girls if you are after game clear only but don;t bother with 3* or less as you will replace them all with 4*+ soon enough.

But if you are a pokemon style player then you will want those gifts just to unlock all the things.