I've thought about this as well, since there are sets all the way down to "Common" items. I tried equipping a full set ages ago and was surprised that I didn't get some boost for having everything matching.

Personally, I think the sets and any full-set bonus should each be only slightly (if at all) biased to a particular specialty. If you want to absolutely maximise your primary stat, you don't get to have the "matching armor" bonus.