Yeah, the balance in the game is definitely wonky. Everything but memento's in the gatcha is pretty much pointless. Even if you're only using basic kitchen/bathroom items and no gifts you're gonna be max affection long far before you have the mementos. It really should only give a message if there's a memento, as those are all that matter.
Still need 5 for Annabelle, 1 for Leilana, and I'm guessing 5 more for Yasu... but I think I've got just about every furniture, most of the outfits, and around 25k lust dust I have no real desire to use to buy anything with... and therein lies another flaw. With you needing premium to make the girls topless/bottomless regardless of affection level, you don't really want to ruin after you lose premium by dressing them up in other things, so all the costume stuff becomes largely pointless.