Those at the top, likely have also much more duel points than you.

As a reminder when you hit level 10 you get +1 duel point then +1 for each 5 level (10, 15, 20 and so on.). So if you try to compete with a level 35 while you are level 12, for example, you will compete with 5 duel points while he has 10 duel points. A gap you will only close with time.