PyData London 2023

Event Driven Machine Learning
06-03, 15:00–15:40 (Europe/London), Minories

This talk focuses on the benefits of using an event-driven approach for machine learning products. We will cover the basics of event-driven architecture for software development and provide examples of how it can be applied for machine learning use cases. The talk will be accompanied by live examples and code for you to follow along, using open source tools such as Apache Kafka, FastAPI and River. By the end of the talk, you'll have a good understanding of the advantages of event-driven architectures, such as improved scalability and responsiveness. If you are a machine learning practitioner interested in exploring this topic, this talk is a great starting point in which we will cover the concepts, tools and common pitfalls of the event driven framework for machine learning products.

This talk is aimed for machine learning practitioners interested in the topic of event driven architecture. Basic knowledge of the machine learning development lifecycle(MLOps) is preferable but not necessary. We will showcase the tools and concepts by implementing an example use case of events coming from users' smartphone applications. All the slides and code used in the talk are available on my GitHub page:

(20 minutes) Main topics covered in the talk:

  • What is event driven software?
    I will present a brief overview of the Event Driven Architecture (EDA) software development approach which enables systems to react to events in real-time. It has recently become popular and widespread by decoupling components and enabling the creation of complex, distributed systems that are more responsive, scalable, and adaptable. The overview will be inspired by the reactive manifesto and it’s corresponding reactive principles.

  • When and why should you implement EDA for machine learning?
    We will discuss how EDA can be a valuable approach for machine learning when real-time, reactive decision-making is required. Additionally, we will show how it can help improve the performance and scalability of machine learning systems by enabling them to process events as they occur, rather than waiting for batch processing. Finally, I will show how it can also enhance the ability of machine learning systems to react to changing conditions, making them more responsive and adaptable to new situations.

  • EDA concepts:
    We will explain concepts and terms such as triggers, brokers, topics, partitions, publishers-subscribers, producers-consumers, stream processing workflows and more.

  • Serverless:
    When used together, serverless tools and event driven systems can be very efficient and powerful. Most of the popular serverless applications are paid services by cloud providers, however we will see how we can achieve a serverless setup which is cloud agnostic and based on open source tools - Kuberentes, Knative and Apach OpenWhisk.

  • Common pitfalls:
    Some of the most common pitfalls when deploying event driven systems can be easily avoided if we prepare for them. These potential issues include the loss of data when an error occurs, congestion at a specific point in the process, version upgrades and synchronous vs. asynchronous event streams. As potential solutions we will look at the delta lake framework and avro file format.

(15 minutes) In the hands on part we will go over a material example of an event driven machine learning product, which uses a Streamlit application, together with a local Kafka cluster and FastAPI application to demonstrate the process of building and updating a Linear Regression using incremental training.

(5 minutes) Q&A

Prior Knowledge Expected

Previous knowledge expected

See also: Slides

Lead Machine Learning Engineer at Zimmer Biomet. London School of Economics graduate with an MSc in Applied Social Data Science. Passionate about using Machine Learning to solve complicated problems. I have experience analysing, researching and building data products in the financial, real estate, transportation and healthcare industries. Curious about (almost) everything and always happy to take on new experiences and challenges. I love finding bugs, especially if they're my own making!