It could differ per building but..
Just upgraded my level 3 fire shrine to 4; 5 hours.
Research Academy went from level 2 to 3; took 1 hour.
It really doesn't matter anyway unless at some point it could take up several days but I don't know if the upgrades even go that far.
I would think after 5 hours comes 10 and after that maybe 15/20 or an entire day.. don't know how quickly it would ramp up.