Unfortunately, you can only slowly accumulate Flower Gems over time if you're not willing to pay for them. I'm at "end game" right now, so maxing out affection, daily bonuses, and 3-medaling all the event missions are the only way to make Flower Gems. Once in a blue moon, a flower gem might drop from gathering but don't count on it.