Don't get from one extreme to another. Castle itself doesn't do much, that's right. BUT...
There is always "but", isn't it?
Every other building you have can't be higher level than your castle. So in case of your base (city? whatever) Castle is your highest priority to level up at least one level above other buildings. Residence and barracks next, then blacksmith and lab, and then whatever you need at the moment.
And yes, you always feel like you don't have enough money.