I only have the 1st barracks, and only need it. Unlike others here, I don't stockpile iron/bronze fodder (my gold is high enough even after AW 20+ units that I can afford to use them even on high level plat/black units. When I don't feel like doing that, I raise pre-CC harem units to 50 for CC so I can complete their scenes.

Even though I am a whale, I am such a whale I will use Gold units as mini-plat armors or AW fodder (and Silver units can be level fodder if they're not crucial CC/AW classes).

TLDR: I burn through my fodder, spirits and dupes quick enough I've never had to go past the 1st barracks (not counting the initial space as a barrack)