Absolutely insane work. So much data you’d think they would come up with a custom solution instead of using the “newest available toolkit” but I understand how much of a mess dealing with that much data is.
Hi Narhem - one of the devs that worked on the migration here. The data volume, and subsequent compute power required to process it, is actually one of the things that led us to Ray (or Ray Core specifically) since it had the distributed computing primitives (tasks and actors) that we needed to build out our envisioned solution with very few compromises. One thing we DIDN'T want to do was just throw another one-liner SQL statement running on a new data processing framework at the problem, since that leads us back to the problems we had with Spark - not enough low-level control for such an important problem.
In short, after evaluating our options, Ray seemed to strike the best balance between the one efficiency extreme of, say, building out custom "compaction-optimized" hardware/clusters, and the other maintainability extreme of just letting the latest managed cloud service run a 1-liner SQL statement for us without ever looking under the hood.
Regardless, I expect both our existing solution and the distributed compute frameworks leveraged to deliver it to continue to evolve over time.
