Yeah, that's the point, gems used for refills are never coming back, and since gems from daily maps don't refresh, the amount we can get for free is limited and numbered (Excluding the really low chance of getting one from gathering). It means they are being used to speed up things due to impatiency. That's what seems silly unless it's reaally necessary due to events or whatsoever.

11x pull gives a 52% chance of getting a 5* or better, with a 6% being a 6* instead which is the only way to upgrade the team outside of event girls. Nonetheless, it has to be noted that going to gacha willy-nilly every time you get 50 stars is also silly. Before doing that you have to take into account what are your chances of getting non repeated 5-6* characters in case you succesfully hit the 52% upper half of the pulls, and if getting doubles is worth it or not. For example, now that there is people with 2 or even 3 5* girls, doing gatcha again has only, roughly a 30% or worse chance to bring something new to the table, so in those cases, sitting on gems until new girls are released might be a better option. Specially if they are still leveling/evolving their 4* and up.

Btw, yeah, if you can, go for the 24/24, i did it al level 30ish with 32K and 20-25K teams and a really nice helper team, so it's not too hard once you find good helpers and rng doesn't troll you with the panels.