Yep, Floor difficulty is not really consistent. I know there is still some hard ones but I don't really remember them. I just remember floor 50 and 80 because I had a hard time on them on the trial castle.
Also, difficulty is really different between hard and normal if you try to get 10k point on each stage. Easy stages can turn into hell if you want to do it damageless.

For what I experienced, Pavonia is sucks on the castle. Actually, I never cleared a harder stage with her. But maybe I just don't know how to use her (that's the same for Tampala. She is supposed to be really good but I never got anything good from her).
For me, higher mode require Adlay & Zinnia + random utility girls. I hope something else work or it would be... meh.
(Obvioulsy I'm speaking of Hard Castle wirth perfect score. Normal Castle is a lot easier and you can use more team variety. Hard Castle is just for extra challenge and personnal satisfaction. It don't gives any specific reward).

I really like the Castle too. Fun and fast battles. I just regret Adlay & Zinnia being that good and almost required. Team building on lower level when they didn't exist was extremly fun. I still remember that damn floor with ultra high def/high ATK boss. I didn't have Aeonium and I couldn't use enough girls for both ATK and DEF debuff and for offense. And I came with hard ATK debuff and pursuit. Pursuit damages were not affected by defense and I had the easiest fight with boss and girls dealing 1 damage and a Sparaxis abusing the pursuit system.