To give a little more detail on how high you want to level buildings:

Castle: however high you have to in order to let you level other buildings. There's little point in getting a castle significantly ahead of your next highest building, however.

Residence: keep pace with your castle level. Residence is much cheaper to level than most other buildings.

Port: additional trade options stop at level 50, so you want to get there, but there's no point in going over this. Like the residence, the port is much cheaper to level than other buildings.

Lab: At least level 30 quickly, and then whatever you need to keep ahead of your highest level lab skill. Lab skills take as long to level as leveling the lab building itself, so there's no need to leave a buffer as with blacksmith.

Temple/Church: At least level 40 quickly for the two main artifacts (convoy carriage and thunder crystal), then level 50 eventually, and above that only if necessary to allow skills to go higher.

Blacksmith: Stay ahead of your highest forged piece of gear. Remember that you can forge gear to have a single piece gain several levels quickly, while gaining several blacksmith levels takes a while. I'd get it at least to 50 before you lay off for a while, but you will need to go higher than 50 if you want to be competitive at endgame.

Barracks: A judgment call on when you want to stop. Once you're getting enough troops that you never have to buy more with copper, a higher level barracks doesn't benefit you at all. Some events do let you gain by burning a ton of accumulated troops, however, such as the current War Prime one.

Dojo: Also a judgment call on when you want to stop. Higher levels are more valuable if you spend koban to unlock extra slots, but remember that regardless, experience comes mostly from special events, not from the dojo. Also, at some point, you end up leveling generals that you have no real intention of ever using because you've already maxed the levels of the generals you want.