While using the silvers you get from completion is quite a waste it's not the end of the world.

All the silvers are (more or less) plentiful from events so you'll eventually get them back.

More importantly, make sure to not class evolve below max level for units you want to keep as those stats are lost forever.

Imo no need to restart, but be careful in the future.