There are no shortcuts. You just need to keep farming.
But since the droprate seems to be ~2%, and you need to do a minimum ~480 Rags per element (60 50% drops per SSR, 4 SSRs), for a total of ~2880 Rags... you should find ~57 Phantoms. From just doing the BARE EXPECTED MINIMUM, and this is without even grinding any Magnas for starting Olympia raids or T4 Souls.
So uh... don't worry about it? You'll MLB them all eventually.