The spirits are indeed tied to a specific skill. They are not like rainbow fairies. They actually have the skill they increase on their stat screen so you can easily see which skill they boost.