A 5* Crit emblem has a truckton more crit on the emblem itself than a 3* crit emblem even before you consider that it gets 5 random mods instead of just 3 that the 3* gets.
Any emblem where the main base of that emblem is for an important stat is something you want to get 5* of if you can, although of course a 5* with horrible random mods is still not going to be as good as a 4* with good mods (the crit on the crit emblem alone or the attack speed on an attack speed emblem will mean that 4* will have to get pretty good mods to beat it though and a 3* will have a lot of trouble beating a bad 5* emblem).
Of course, that's just for certain emblems. Crit, Crit Damage, Attack Speed, Evasion, and to a lesser extent +HP are the type of emblems that you really want to get 5*s of. A 5* accuracy or penetration emblem won't make that much of a difference (penetration can't reduce armour below 0 so you usually are fine with just a 4*), and a 5* movement speed or rotation emblem is usually hot garbage no matter what the mods are.