Building a serverless computation environment with PythonView
Born and raised in Zimbabwe, currently living in the UK. Passionate about excellent software, design and good experiences. Spent 5 years at CMR Surgical doing electronics and software for an amazing surgical robot.
Building a serverless computation environment with Python
This talk will be a whistle-stop tour of a virtual machine, compiler, and DSL for orchestrating Python computation on serverless cloud technology. The goal is to be able to take existing code and get it running in a very scalable way on the cloud, with as little effort as possible. It’s easy to deploy one AWS lambda function (eg with the well known Serverless Framework), but it’s much harder to write several functions and pass data between them. The Teal project started as an experiment to see whether it’d be feasible to simplify this without resorting to simple “workflow” or “task” based approaches. The results have been promising.
There will be a demo (!) - build, test and deploy a non-trivial new data pipeline, mostly written in python, in less than 5 minutes.
Something that might have taken several hours before, or longer if you didn’t have orchestrator or task-runner infrastructure set up already. We may also cover how this makes development and CI/CD easier. Depending on audience & time, we may dive deeper into the implementation approach, and what Python features made it a great experience.