Ilan Ginzburg - Search in the Cloud: separation of compute and storage

The traditional architecture of a search cluster, with nodes providing compute and persistent storage, ends up being more expensive to run than it should, especially in the context of public cloud and when dealing with unpredictable traffic patterns/load.
This presentation is about modifying the implementation of SolrCloud so a single copy of each index is stored on shared storage (for example S3) and accessed by all nodes for reads and updates. The nodes are then stateless, which means any one (or all) of them can be turned off at any time without data loss. New nodes added to the cluster get their data from the shared storage and not from other nodes. Note that many aspects of this new architecture apply to other search clusters, and no prior knowledge of SolrCloud is assumed.
Such an approach is a good match for running Search on a public cloud infrastructure, as it allows for compute and storage to be scaled independently and decreases the overhead of bringing a new node up in the cluster.
The challenges faced in the process of separating compute and storage will be detailed. The hard problems concern updates (indexing), and protecting the shared storage from conflicting concurrent updates, preventing data loss or overwrites yet allowing concurrent updates. Tradeoffs for integrating such a large conceptual change into an existing system will also be discussed.
The existing code has limitations that will be presented, as well as a high level design and plans for lifting them.
The current code is available (see SIP-20 cwiki.apache.o... for code and documentation).
The aim is to eventually merge these changes into the main Solr branch.
Speaker: Ilan Ginzburg
More: 2024.berlinbuz...
###
Follow us on Social Media and join the Community!
Mastodon: floss.social/@...
LinkedIn: / berlin-buzzwords
Instagram: / berlinbuzzwords
Website: 2024.berlinbuz...
Berlin Buzzwords is an event by Plain Schwarz - plainschwarz.com