Thread: Let's talk about snatch

How many total Snatch/Snatch-less Ultimate runs would need to be logged in order to be statistically relevant?

Originally Posted by Skulkraken

How many total Snatch/Snatch-less Ultimate runs would need to be logged in order to be statistically relevant?

Assuming a 5% drop rate increase, we'd need about 100 chests of a given rarity in each Snatch group (Snatch, no Snatch). To observe a smaller effect, we'd need more.
Sample size calculator

Ideally, you'd then want to repeat that a few times. That's why I recommended pooling data from multiple users.

...I see.

Wouldn't it be fine to start with just copper chests, then? Ultimate always drops 20+ of them, so it should be feasible even for one player to collect enough of them to do an analysis in just 5 or so runs...or am I misunderstanding what you said?

Originally Posted by Skulkraken

...I see.

Wouldn't it be fine to start with just copper chests, then? Ultimate always drops 20+ of them, so it should be feasible even for one player to collect enough of them to do an analysis in just 5 or so runs...or am I misunderstanding what you said?

Copper chests are a good place to start. But since they drop from trash mobs, that complicates the experiment. You'll have to make sure you land X stacks of Snatch average on each mob (and a minimum of X stacks on the boss). My initial power estimate assumed 5 stacks. We can lower that number, but then you'll have to collect more chests (e.g., 300 for 3 stacks, 700 for 2 stacks). Meanwhile, the control group will do all runs without Snatch. For each group, record the number of chests of each rarity that drop in each run (just record the final tally the game gives you before you hit the next button) until a total of 100 chests of the rarity of interest are collected (more if less than 5 stacks of Snatch per mob). We can then do a t-test on the series of drops per run in each group.

As long as each person uses the same methodology, numbers can be pooled to achieve these targets.

Alright, then. I'll give it a go a little bit later. Whales, can you log your Snatch runs? I'll log some no-Snatch runs for use as the control group.

Now that I have some pilot data, I can give more exact sample number requirements. Here's the number of runs required to spot a difference in bronze chests assuming X stacks of Snatch per mob on average (minimum X stacks on boss). This assumes an effect of Snatch of at least 1% per stack.
5 stacks (5% difference): 5
3 stacks (3% difference): 12
2 stacks (2% difference): 27
1 stack (1% difference): 105

I currently have 13 control runs recorded (no Snatch).

- - - Updated - - -

My Snatch data is available in my Kamihime Project spreadsheet on the Snatch tab. I reached my quota for the event and will likely not be running much more. Do note that if the Snatch group data isn't populated before the end of the event, the opportunity to use this data for statistical analysis will be lost. Intriguingly, my first run using Snatch gave me more bronze chests of the 13 runs without, and matched the maximum of silver chests from those runs, suggesting a readily observable effect. The run also took like 50% longer...

Okay, 100 (give or take) copper chests:

Run 1 - 21 copper, 5 silver, 3 gold
Run 2 - 24 c, 5 s, 3 g
Run 3 - 21 c, 6 s, 3 g
Run 4 - 18 c, 5 s, 3 g
Run 5 - 23 c, 5 s, 3 g