I wasn't saying that you could spend flower gems to raise affection on girls. I was saying that you *got* flower gems for raising affection on girls. And so you could get 20 flower gems for max 1st affection on girls. Or you could use all that material and max 2nd affection on 1 girl (and get 2 flower gem reward, iirc).