I maxed affection on all my C, UC, and Rs, then sacrificed/fused most of them. There are a few worth keeping, several UCs with bonuses for barracks, one for residence, then there are a few Rs that boost some of the UR/SSR teams, and you'll want at least 2 Cs for quests. If you sacrifice a R you end up wanting back later it's a kick in the nuts, but I don't see any other way to do it. Later on when your castle is really high and you have research that apparently boosts slots further you can start collecting everything, early on you just need to dump it all so you can keep summoning and get better generals. I don't imagine it will be that hard to get back one of every C/UC/R later on.

Inaba Ittetsu (UC) is needed for a couple stars so hold onto her if you get her. That's the only card specifically required by any quests I've seen so far.