If you have room, you should keep those and give them all second affection when possible so you get the gems and H Scenes. Then its up to you. Some collect all and others don't.

I personally keep the top 40(or more) of each color.