Yeah, what phob said is pretty much it. Per girl you can get 3 gems. It really just depends if you want to have one copy of each girl in your inventory. However, you might need them for Nation Battles where you are forced to use knights from single nation. Also, I'm not sure if you're going to have trouble with inventory space this early in the game.