The reason why having more troops is important is not just that it takes more damage to kill you. It's also that adding troops adds attack and defense, too. If you're equipping accessories that are lower level forging than your weapons or armors, then you're doing it wrong unless you think you have endgame-worthy gear in the other slots but only junk accessories that you want to replace as soon as possible.