You're not alone in that feeling. 5/5 Olivie and Karma are clear stand outs due to combining high magic damage and the ability to skill tank things like SS3 Black Oni for brief periods but most other duelists have to wait until they are Awakened to start really differentiating themselves. Of the available units my dream would be running Olivie & Lyla as my main duelists with Thetis pinch hitting on paralysis maps, but luckily there's ways to work around the absence.