11-09, 15:30–16:15 (America/New_York), Central Park West (6th floor)
Deploying Dask
Dask is a framework for parallel computing in Python.
It's great, until you need to set it up.
Kubernetes? Cloud? HPC? SSH? YARN/Hadoop even?
What's the right deployment technology to choose?
After you set it up a new set of problems arise:
- How do you install software across the cluster?
- How do you secure network access?
- How do you access secure data that needs credentials?
- How do you track who uses it and constrain costs?
- When things break, how do you track them down?
There exist solutions to these problems in open source packages like dask-kubernetes, helm charts, dask-cloudprovider, and dask-gateway, as well as commercially supported products like Coiled, Saturn, QHub, AWS EMR, and GCP Dataproc. How do we choose?
This talk describes the problem faced by people trying to deploy any distributed computing system, and tries to construct a framework to help them make decisions on how to deploy.
Deploying Dask
Dask is a framework for parallel computing in Python.
It's great, until you need to set it up.
Kubernetes? Cloud? HPC? SSH? YARN/Hadoop even?
What's the right deployment technology to choose?
After you set it up a new set of problems arise:
- How do you install software across the cluster?
- How do you secure network access?
- How do you access secure data that needs credentials?
- How do you track who uses it and constrain costs?
- When things break, how do you track them down?
There exist solutions to these problems in open source packages like dask-kubernetes, helm charts, dask-cloudprovider, and dask-gateway, as well as commercially supported products like Coiled, Saturn, QHub, AWS EMR, and GCP Dataproc. How do we choose?
This talk describes the problem faced by people trying to deploy any distributed computing system, and tries to construct a framework to help them make decisions on how to deploy.
No previous knowledge expected
Matthew is an open source software developer in the PyData ecosystem. He primarily works on Dask, a library for parallel computing in Python. Matthew worked for Anaconda and NVIDIA before starting a company, Coiled with a mission to enable scalable computing for the Python community.