Because sometimes you just need to do it yourself, with Python

TL;DR: This post describes a basic design that will allow you to distribute [large] tasks to multiple workers running in Kubernetes to be run in parallel. It uses containers that run a listener service (that you write). You deploy these containers using a Kubernetes Job workload controller. You then send your various tasks to all of the containers with some client code. Your listener app deserializes the work, runs it, and returns the output. This is the architecture in a nutshell.

The tips and details I provide will be using Python, but the general design could work in any language…

John Raines

John is a software engineer who primarily works in data science platform design.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store