Originally Posted by Corintis

I don't even know how to spend Flower Gems to raise affection

The only way you could is by burning them for stam to run the Searching for Gifts mission then forging those gifts, which I really do not recommend unless a) you're swimming in FGs and b) so desperate to get a girl with full 2nd affection that you can't wait until you can just buy the gifts in a later update.

But as everyone else has said thus far, just carry on and ignore the second affection for now.