Apart from getting almost nothing the first half year I played, I cannot complain about my luck all that much.
So far I've gotten:
Dark Lord: all three
Ninja Spy: Solveig <-- this is my biggest foe so far
Gruntt: all three
Edwarde: all three
Donatien: all three
Silvanus: Sylvia and Leeditt
Bremen: Stephanie and Marina
(so just 4 missing)
And from the events I've gotten:
Halloween: Halloween Juliette
Christmas: Christmas Bunny and Winter Agate
Valentine: Val Red Battler and Val Abraƫl
Spring Break: ... <-- this was the only shitty event for me so far
Easter: Hari and Rabbi
Festival: Festival Solveig and Festival Bonny
Music: Metal Noemy (festival is still going on)
So to answer your question, Ninja Spy has been the most annoying so far.
The thing is, you will get your girls eventually.
Like I said, I had to wait a long time before getting lucky.
I got just 2 girls in the first half year or so.
I have this explanation: the girls want a strong man, so as long as you do not have a strong harem, they are hesitant to join your harem