Level Up in Python with Dependency Inversion and Entry Points

Photo by Tim Gouw on Unsplash

Python programmers don’t use Dependency Inversion (DI) enough. There are a number of reasons why not:

• It’s tricky — it inverts certain intuitive understandings of code flow.
• Sometimes people do dependency injection and think they’ve done Dependency Inversion.
• The benefits of DI are easily visible in mature codebases, but implementing DI in new code adds a level of abstraction that many programmers feel is cumbersome. It gets YAGNI’d, when it shouldn’t.

But dependency Inversion DI could save the code you’re writing today from being obsolete in 2 years.

What’s more, Python has a fantastic native feature called Entry Points (supported by most packaging tools) that allows you to get a lot of extensibility out of Dependency Inversion, easily.

So here’s what I plan to cover in this post:

• What is Dependency Inversion (and what does it look like in Python)?
• What are Python Entry Points?
• How Do These Two Things Go Together?

If you’re a Dependency Inversion Champ and feel like you have a SOLID handle on the concept (get it?), feel free to skim down to the Entry Points section.

What is Dependency Inversion?

Definitions

There’s a standard definition of the Dependency Inversion principle. It’s absolutely correct, but when you’re first learning about DI, it feels cryptic. Lets get it out of the way:

High level objects should not depend on low level objects. Both should depend on abstractions.

Let me borrow from the conceptual world that generated that definition to give you a different but compatible one:

The core logic of your code should define and use only the interface it wants to interact with.

A Cute Example To Set Us Up

The core logic of your code is the part that defines the special things you want your software to do in terms of high-level abstractions. Remember this viral code meme joking about how they managed to keep drones from crashing into each other in the Olympics opening ceremony?

// JavaScript

if (goingToCrash()){
    dont();
}

The level of abstraction makes us chuckle, but it’s not actually terrible. It really is important to separate the code that expresses our core logic from the details of how that logic is implemented. Let me re-imagine it in Python:

def prevent_vehicle_from_crashing(vehicle: VehicleBase):
    if vehicle.going_to_crash():
        vehicle.dont_crash()

From this code snippet, we know a few things:

• This code interacts with a vehicle object.
• The vehicle object has an interface that includes methods going_to_crash and dont_crash.
• This code is concerned with not letting vehicles crash.

This is where our working definition of Dependency Inversion will come into play: the core logic of your code should define and use only the interface it wants to interact with. That means that there’s a lot that our core logic doesn’t know. Most importantly,

The code doesn’t know what classes vehicle could possibly be.

All the concrete classes we can imagine — Drone, Car, Boat, etc. — will have all sorts of other methods and capabilities unique to the specific sort of vehicle that class represents. But our core logic doesn’t want any of that information. It simply wants to prevent crashes with the most straightforward interface possible. It works for any class that implements this interface, regardless of whether the implementation uses Numpy or some custom library you wrote in C.

From this, we can see that our code above is satisfying part of our definition of Dependency Inversion. It is using only the interface that it wants to interact with.

In order to satisfy our full definition, our code should also define this interface. This is actually the heart of Dependency Inversion — that will become more clear as we go along. Let’s do it. Here’s our new core logic module:

class VehicleBase:

    def going_to_crash(self):
        raise NotImplementedError

    def dont_crash(self):
        raise NotImplementedError


def prevent_vehicle_from_crashing(vehicle: VehicleBase):
    if vehicle.going_to_crash():
        vehicle.dont_crash()

And Now the Inversion

Great! Now let’s go use our new code to prevent vehicle crashes! Just kidding. You can’t. It’s all abstract.

So what good is it? Well, for one thing, notice that there’s no import block in the module above. Our core logic has no dependencies. Instead, when it comes time to write concrete Vehicle classes in some other part of the code base, those classes will depend on our core logic module for the definitions of the interfaces that they must implement.

Here’s what that will look like:

from vehicle_crash_prevention import VehicleBase

class Drone(VehicleBase):

    def going_to_crash(self):
        # Check Drone's state to see if a crash is immanent

    def dont_crash(self):
        # take Drone-specific actions to avoid a crash

This has the effect of inverting the dependency flow in the opposite direction of the operational/information flow. Our core logic acts on things external to it — it acts on concrete vehicle objects. In a more complex implementation, it may even pass information into those external entities. The operational/informational flow is an arrow outward from our core logic.

But in terms of dependencies, it doesn’t depend on concrete information about any of these things in order to do what it does. Instead, all of those concrete things depend on it to tell them how they should provide access to the various behaviors it wants to perform.

Let me give a couple of reasons that this dependency flow is so important. Remember at the top, I mentioned that Dependency Inversion can stop your code from being obsolete in a couple of years? That’s because…

• DI protects your core logic. The core logic will be slow to change, and that is kept separate from concrete implementation details that will change within a couple of years. The core logic doesn’t import tons of external dependencies that you have no control over. Instead, you will write outer layers of code that adapt the external libraries to this core logic.
• DI makes your code base extensible. You can implement your core logic as abstractions, and then implement each new vehicle concrete class as a plugin. Heck, you don’t even need to know about all the weird vehicles people are going to dream up and then keep from crashing by using your code. Anyone else’s code that implements the interfaces you define will work with your code, without needing any changes from your code. That’s how extensible DI makes your code.

Quick Note(s) on Real Life Code

In a real codebase, you’ll be breaking your class definitions into separate modules. And in almost all libraries and applications, you will need some concrete classes that do have outside dependencies. In those cases, the collection of modules that define your core logic should still be a fairly small, fairly closed system that does not depend on a ton of external modules. The concrete classes live outside this core logic, using the abstractions it provides, while also using external dependencies.

What Are Python Entry Points?

Ok, let’s shift gears for a second and talk about Python Entry Points. Python Entry Points are a specific kind of metadata that you include in your packaged code. With Entry Points, you can separate core logic into its own package, and then write new packages that provide plugins for your core logic.

(A quick heads-up, if you want to actually use Python Entry Points, you have to become familiar with packaging your python code. That’s beyond the scope of this article, but it’s a pretty important skill. Here are links to a few packaging tools, setuptools, poetry, & hatch, which each provide documentation and tutorials.)

Exposing the Entry Point in the Core Logic module

Using our example above, our core logic might include Entry Point metadata that exposes one plugin entry point for Vehicle plugins. We do that by adding some metadata in our pyproject.toml. If you’re using setuptools as your packaging tool, that looks like adding the following lines:

# pyproject.toml for setuptools

[project.entry-points."no_crashes.vehicle"]
base = "vehicle_crash_prevention.main:VehicleBase"

Some notes on the configuration code above:

• I’ve called this entry point "no_crashes.vehicle". You could name it whatever you want (e.g. just "vehicle” if you want). Using this format with the dot (.) and a prefix just allows me to namespace my plugin a bit and avoid naming collisions, but it’s optional.
• For exposing the plugin, my convention is to register the base class as if it were a valid plugin. This is done by providing the importpath.to.thing:ThingToImport as the plugin value. Internally, Python’s Entry Points process will use this path to do essentially the following if the Entry Point is actually used and loaded:

from importpath.to.thing import ThingToImport

Registering Plugins from External Packages

Now, maybe you’ve got a package called boats that provides a bunch of concrete Vehicle implementations for uh… boats? That means that this new package includes some code that kind of looks like:

# main.py

from vehicle_crash_prevention.main import VehicleBase

# Assume we've implemented these:

class Dinghy(VehicleBase):
    pass

class CruiseShip(VehicleBase):
    pass

Your pyproject.toml for this package would include the following:

# pyproject.toml for setuptools

[project.entry-points."no_crashes.vehicle"]
dinghy = "boats.main:Dinghy"
cruise_ship = "boats.main:CruiseShip"

Your package metadata is now configured for these classes to be used as plugins to your vehicle_crash_prevention package. The header name indicates that when package metadata is parsed, these plugins will be grouped with the one exposed by our core logic package. We provide a name for this plugin, boats, and then also the import path to the class we’re registering. For example the definition above indicates that we’ve got a plugin that we’re naming dinghy. It will register the Dinghy class as a plugin, and the Dinghy class lives in the boats.main module.

There’s just one more step to implement before we can actually use these plugins in an application. Next section, please…

How Do These Two Things Go Together?

Dependency Inversion with plugins allows for the following application dependency structure:

Notice a few things.

• The dependency graph is acyclic. Dependency lines terminate in either 1) our core logic package or 2) Python’s standard library Entry Points (i.e. the Python runtime). Our core logic does not depend on any of our other code, despite the fact that it’s going to be controlling objects passed to it by the application.
• What’s more, our application is decoupled from all of our plugins by the Entry Points interface, which means the application itself doesn’t depend on concrete implementations either! This decoupling will allow for tons of flexibility as requirements and environments change.

Now let’s apply this to our ongoing example:

I mentioned above that there was one more implementation step before we could use our plugins in an application. Our Crash Preventer application needs to load any available plugins for the vehicle entry point. Plugins for this entry point are available to be loaded if packages that register those plugins have been installed in the environment. That is, if you’ve installed the rockets package, then its plugin called falcon9 will be available for use. Let’s look at how the standard library allows us to load plugins in a way that’s decoupled from the plugin packages themselves.

Remember from our initial section on DI that our core logic package exposes an entry point named "no_crashes.vehicle". We can access and load all of the plugins defined for this entry point using the standard library’s importlib.metadata module like so:

from importlib.metadata import entry_points

"""
# for python 3.6 to 3.9: 
vehicle_entry_points = entry_points()["no_crashes.vehicle"]
"""

# python 3.10+
vehicle_entry_points = entry_points(group="no_crashes.vehicle")

vehicle_classes = {
    v.name: v.load() 
    for v in vehicle_entry_points 
    if v.name != "base"  # ignore the VehicleBase class
}

Now we’ve loaded all the Vehicle classes that have been installed in our environment. Let’s look at how we might use them.

Because we’re engineers and professionals, our Universal Vehicle Crash Preventer application is going to have a config that declares vehicles that we expect our application to manage:

vehicles:
  - type: dinghy
  - type: falcon9
  - type: steam_engine

Notice that this is the only place that information about specific vehicle types lives in our application — in a data layer. Our application itself will be agnostic of types and number of vehicles it manages. With this config loaded as a dict, let’s instantiate all our vehicle plugin classes and call our core logic code:

from vehicle_crash_prevention.main import prevent_vehicle_from_crashing

config: dict = load_config()
vehicles = [
    vehicle_classes[spec['type']]()
    for spec in config['vehicles']
]

while True:
    for vehicle in vehicles:
        prevent_vehicle_from_crashing(vehicle)    

Conclusion

Dependency Inversion is one of the most underused principles in Python programming, but it shouldn’t be! By protecting the core logic of your code from implementation details, you’re able to extend your code’s capability without being worried that the core logic is going to break every time you need to change something in the periphery of your codebase. And when you use Entry Points, the extensibility becomes as easy as just installing a new plugin and adding to a config file! I hope you give it a try.