Welcome to the 22nd London Information Retrieval & AI Meetup, a free evening event aimed at Information Retrieval and AI enthusiasts and professionals who are curious to explore and discuss the latest trends in the field.
The second talk is from Daniele Antuzi, R&D Software Engineer @ Sease.
Title: "Blazing-Fast Serverless MapReduce Indexer for Apache Solr"
Abstract: "Indexing data from databases to Apache Solr has always been an open problem: for a while, the data import handler was used even if it was not recommended for production environments. Traditional indexing processes often encounter scalability challenges, especially with large datasets.
In this talk, we explore the architecture and implementation of a serverless MapReduce indexer designed for Apache Solr but extendable to any search engine. By embracing a serverless approach, we can take advantage of the elasticity and scalability offered by cloud services like AWS Lambda, enabling efficient indexing without needing to manage infrastructure.
We dig into the principles of MapReduce, a programming model for processing large datasets, and discuss how it can be adapted for indexing documents into Apache Solr. Using AWS Step Functions to orchestrate multiple Lambdas, we demonstrate how to distribute indexing tasks across multiple resources, achieving parallel processing and significantly reducing indexing times.
Through practical examples, we address key considerations such as data partitioning, fault tolerance, concurrency, and cost.
We also cover integration points with other AWS services such as Amazon S3 for data storage and retrieval, as well as DynamoDB for distributed lock between the lambda instances."
If you are willing to attend the next London Information Retrieval & AI Meetup, don't forget to join our group: bit.ly/2IjSBeX
We are also accepting talks for the next meetups. If you have any talk you would like to propose, feel free to send us an abstract at talk@sease.io.
**********************************
Hosted by Sease: sease.io