Stop Training Models in DEV. Train Them in LAB.

Photo by Raghav Bhasin on Unsplash

Congratulations, ML professionals of all kinds! Only 10 years after Andrew Ng taught everyone about gradient descent, our respectable profession has almost entirely moved away from using the PROD environment for everything. Pat yourself on the back.

Now shame on you! Because I glanced over your shoulder just now and saw you training a model in DEV. There’s a better way, people, and it’s called LAB.

So here’s what I want to tell you about:

• Environments — what they are.
• How Machine Learning has two very distinct concepts of a development environment, DEV and LAB, that should not be confused (or share the same name).
• How to organize your team’s work between DEV and LAB, and the implications for other environments.

Environments

What are Environments?

Let’s talk for a minute about Environments. First off, let me provide a dense definition of Environment and then unpack it a bit. For the purpose of this blog, an Environment is a runtime that is configured to use a specific group of infrastructure resources for a particular use case. (This definition mashes a few different concepts together, namely runtime, resources, and use case. Here and here are some useful blog…