11-10, 11:45–12:30 (America/New_York), Winter Garden (5th floor)
Workflow engines are widely used, but most people don’t know how they work under the hood. This talk will review how a team of engineers implemented a workflow engine from scratch in Python.
The talk will cover the motivation for building a new workflow engine and why existing solutions did not fit the use case. Next, it will talk about the architecture and implementation of the engine, including queuing, managing workers, asynchronous execution, task polling, deployment, and monitoring.
It will show code examples that use popular Python libraries such as Pydantic and FastAPI in the implementation of the engine.
In this talk, you will learn what a workflow engine is used for and how this particular one is architected. You’ll learn how to use Python libraries to handle queuing, serialization, asynchronous execution, and other aspects of a workflow engine – as well as understand other concepts related to at most once semantics, idempotency, and replayable workflows.
This talk will help clarify what goes into a workflow engine and how to build one using a familiar Python tech stack.
Previous knowledge expected
Sanjay Siddhanti joined AKASA as an early engineer in 2019 and currently serves as director of engineering. He has a passion for working on software to help people have a better experience with healthcare. His teams focus on AKASA’s AI-driven automation platform and data engineering problems. He has a B.S. in computer science and M.S. in biomedical informatics from Stanford University.