I often thought the same thing. You only see pictures of your harem when you increase their affection level and that's not always much. The ones we get from quests we get to see during the quest, but most of the drops we do not. Only exception I can think of to that was during the Gems kingdom quest you got to go with Shehermazond.