A Simple Plugin Framework

by Marty Alchin on January 10, 2008 about Django

Since getting involved with Python, I’ve read a few discussions about Zope, and though I haven’t used it, I do enjoy reading articles about it, to see how other people approach common problems. In particular, a recent Satchmo discussion pointed me to an article about using Zope3 interfaces to essentially manage plugins. I knew Trac did this, as I had tried to write a Trac plugin a while back, but I hadn’t really seen a good description of how the process actually works until now (thus my Trac plugin was doomed from the start).

I must say, I’m not impressed with the process.

To my understanding, having worked a bit with interfaces in Java and a little bit in PHP5, interfaces are a form of design by contract, essentially using a programming language itself to verify that the programmer is doing things correctly. This can be done at compile-time (as in Java) or at run-time (as in Zope), but the effect is the same: if you implement an interface and you don’t do so correctly, you’ll get all sorts of errors indicating this. It really has nothing to do with plugins at all.

As an aside, it seems to me like interfaces (and design by contract in general) exist for the sole purpose of making sure that programmers don’t make mistakes. They don’t add functionality or make anyone’s job easier, they just protect against programmer incompetence. And personally, I find it offensive to think that my programming language or framework assumes me to be incompetent. I may be misunderstanding, of course, but that’s certainly how it comes across to me, and I don’t want any part of it. I much prefer duck typing, though I fully understand that this is all a matter of personal opinion.

Of course, Zope provides a component system as well, which is designed for plugins, but a natural problem occurs: it relies on interfaces to get the job done. This is unfortunate, as it incurs a lot of extra overhead that’s intended to serve a completely unrelated purpose. So, as was the case in the Stereoplex article linked above, the programmer is often required to maintain a significant amount of bookkeeping code just to make the system work, which takes time and energy away from the real work to be done: dealing with plugins. And yes, I realize that the bookkeeping code here is less than 10 lines, but why need any at all?

The bare essentials

So, what does it really take to implement a plugin “architecture”? Outside the framework, when implementing plugins themselves, there are three things that are really necessary:

  1. A way to declare a mount point for plugins. Since plugins are an example of loose coupling, there needs to be a neutral location, somewhere between the plugins and the code that uses them, that each side of the system can look at, without having to know the details of the other side. Trac calls this is an “extension point”.

  2. A way to register a plugin at a particular mount point. Since internal code can’t (or at the very least, shouldn’t have to) look around to find plugins that might work for it, there needs to be a way for plugins to announce their presence. This allows the guts of the system to be blissfully ignorant of where the plugins come from; again, it only needs to care about the mount point.

  3. A way to retrieve the plugins that have been registered. Once the plugins have done their thing at the mount point, the rest of the system needs to be able to iterate over the installed plugins and use them according to its need.

That may seem like an incredibly complicated task, and it certainly could be, as evidenced by the Zope implementation of the above requirements. But it can actually be done in just 6 lines of actual code. This may seem impossible, but the following code successfully fulfills all three of the above requirements.

class PluginMount(type):
    def __init__(cls, name, bases, attrs):
        if not hasattr(cls, 'plugins'):
            # This branch only executes when processing the mount point itself.
            # So, since this is a new plugin type, not an implementation, this
            # class shouldn't be registered as a plugin. Instead, it sets up a
            # list where plugins can be registered later.
            cls.plugins = []
        else:
            # This must be a plugin implementation, which should be registered.
            # Simply appending it to the list is all that's needed to keep
            # track of it later.
            cls.plugins.append(cls)

Yep, that’s it. All that’s left from the framework side is documentation, as the above code doesn’t make it immediately clear how you’re supposed to achieve the three core requirements. In the following sections, I’ll be reimplementing the system illustrated in the Stereoplex article, in hopes of an apples-to-apples comparison.

Declaring a mount point

Since the above class subclasses type, it can be used as a metaclass. Exactly what that means internally is beyond the scope of this particular article (though I keep intending to write it up properly), so I’ll just focus on how to use it.

class ActionProvider:
    """
    Mount point for plugins which refer to actions that can be performed.

    Plugins implementing this reference should provide the following attributes:

    ========  ========================================================
    title     The text to be displayed, describing the action

    url       The URL to the view where the action will be carried out

    selected  Boolean indicating whether the action is the one
              currently being performed
    ========  ========================================================
    """
    __metaclass__ = PluginMount

That may look daunting, but the vast majority — and certainly the most important part — is documentation. Believe it or not, that last line is all it takes to do the real work. ActionProvider can now serve as a mount point for plugins that provide actions. As explained in the next section, individual plugins will subclass this, so if you want to provide any default attributes or helper methods, you can do so on this class, just like you would on any other class. They’ll get inherited by each individual plugin in turn, which, depending on what you’re trying to accomplish, could make the development of individual plugins much easier.

Registering a plugin

Now that we have a mount point, we can start stacking plugins onto it. As mentioned above, individual plugins will subclass the mount point. Because that also means inheriting the metaclass, the act of subclassing alone will suffice as plugin registration. Of course, the goal is to have plugins actually do something, so there would be more to it than just defining a base class, but the point is that the entire contents of the class declaration can be specific to the plugin being written. The plugin framework itself has absolutely no expectation for how you build the class, allowing maximum flexibility. Duck typing at its finest.

class Overview(ActionProvider):
    title = 'Overview'
    view = task_detail

    def __init__(self, request, *args, **kwargs):
        self.url =  reverse(self.view, args=args, kwargs=kwargs)  
        self.selected = request.META['PATH_INFO'] == self.url

class Milestones(ActionProvider):
    title = 'Milestones'
    view = task_detail

    def __init__(self, request, *args, **kwargs):
        self.url =  reverse(self.view, args=args, kwargs=kwargs)  
        self.selected = request.META['PATH_INFO'] == self.url

Since I’ve eliminated the need for the TaskActions layer described in the original example, I’ve created two separate plugin classes, one for each action that should be made available. To me, this seems an even better approach than the Zope-oriented example, because there’s no longer some middle-man class that has to know about all available plugins. Instead, plugins can located anywhere in any application, as long as the module containing them gets imported at some point. That really is the only requirement. Once the first plugin is imported, it’ll import the mount point, which will initialize it if it isn’t already, and when Python prepares the plugin class, it’ll be registered and immediately available for use. No muss, no fuss. Just define your plugin and run with it.

Of course, both classes now share an identical constructor, so this is a prime candidate to be moved into the plugin mount class. Just make sure to update the ActionProvider‘s docstring, since plugins would now have to supply title and view, with the rest being handled automatically for all plugins. Or, if that doesn’t seem appropriate for your taste, you could create a separate class to hold the constructor, then use multiple inheritance to pull it into the plugin classes along with the mount class. Either way, any plugin that wishes to customize the behavior provided by the constructor may still do so simply by defining the method in its own class. Remember folks, this is all just standard Python. The usual rules apply.

Here’s how ActionProvider and its plugins would look after factoring out the common code.

class ActionProvider:
    """
    Mount point for plugins which refer to actions that can be performed.

    Plugins implementing this reference should provide the following attributes:

    =====  ============================================================
    title  The text to be displayed, describing the action

    view   The view which will perform the action, as a callable object
    =====  ============================================================
    """
    __metaclass__ = PluginMount

    def __init__(self, request, *args, **kwargs):
        self.url =  reverse(self.view, args=args, kwargs=kwargs)  
        self.selected = request.META['PATH_INFO'] == self.url

class Overview(ActionProvider):
    title = 'Overview'
    view = task_detail

class Milestones(ActionProvider):
    title = 'Milestones'
    view = task_detail

But if you’re providing a mount point and expecting people to write plugins for it, how are you supposed to know if the plugin is written correctly? Documentation. As shown in the examples, just document how your code expects the plugin to behave, and that’s all. It’s the job of the plugin developer to make sure that the plugin is written correctly, and that it conforms to the documented requirements. That’s the essence of duck typing: you provide an object that supports a particular “protocol” (a set of attributes or functions that behave in an expected manner), and anything that uses that protocol will be able to use that object. If it walks like a duck and talks like a duck, just use it as a duck.

Utilizing available plugins

That brings us to the third requirement. Now that we have a couple plugins available to our code, how do we get at them? The Zope example uses some component.getMultiAdapter nonsense, but if you look closely at the mount point class, you’ll see that it has a plugins attribute, which — wonder of wonders — is a list of available plugins, ready and waiting to be iterated. Wherever you’d like to use these plugins, just import the mount point, iterate over it, and have your way with them. The following template tag is identical to the Zope example, but with the plugin access technique changed to use the mount point instead.

from django.template import Library
register = Library()

@register.inclusion_tag('templatetags/actions.html', takes_context=True)
def actions(context):
    obj_name = context.get('obj_name', 'object')
    obj = context.get(obj_name)
    request = context['request']
    actions = [p(obj, request) for p in ActionProvider.plugins]
    return {'actions': actions}

Only the last two lines were changed, and you may notice that it completely removes the need to call any framework-wide method. The mount point already knows about its plugins, so that’s all you need. Using a list comprehension, each plugin is instantiated with the appropriate data, then the list is passed into the template, which can be used as is, unchanged.

Providing a utility method

Now, I don’t see a need to make this “framework” any more complicated than it already is, but to truly make an apples-to-apples comparison, the Zope example doesn’t require the programmer to assemble the list of instantiated plugins explicitly wherever needed. Instead, a method call with the appropriate options will do the trick. So, if this is your cup of tea, just add the following method to PluginMount.

    def get_plugins(cls, *args, **kwargs):
        return [p(*args, **kwargs) for p in cls.plugins]

This uses dynamic arguments to create a pass-through, blindly sending along whatever arguments were sent to it. So, just update the template tag to pass in some arguments, and you’re all set.

    actions = ActionProvider.get_plugins(obj, request)

Using plugins on a model

While the above example provides an easy way to manage plugins, many situations may expect to have plugins available as an attribute of a model class. That way, any time you have an instance of that model, you also have all the plugins that could be applied to it. The simplest way to do this would be to simply assign the list of plugins as an attribute of the model. Since lists are mutable, you can assign it to your model even before any plugins are found, and the list will update according to new plugin registrations.

class Article(models.Model):
    validators = ValidationProvider.plugins

    # ... fields and methods

    def validate(self):
        for validator in self.validators:
            validator(self).validate()

If you really wanted to get fancy with this, though, you could use a descriptor to automate some of this. Since the validation plugins this code expects will always take a model instance as the first arguments, and descriptors automatically receive the instance without any extra work on your part, it’s a natural fit. Consider the following descriptor class, which manages the plugins and passes the model instance in as the first parameter for you.

class ModelPlugins(object):
    def __init__(self, mount):
        self.mount = mount

    def __get__(self, instance, owner):
        return [p(instance) for p in self.mount.plugins]

Now our User object can pass the mount point to the descriptor and use that instead, to make the model code much simpler.

class Article(models.Model):
    validators = ModelPlugins(ValidationProvider)

    # ... fields and methods

    def validate(self):
        for validator in self.validators:
            validator.validate()

This does change the semantics a bit, though, since now each item in the list will be an instance of the plugin, rather than the class itself. This also means that you’re no longer able to pass in additional arguments as it is. Both of these can be solved by using curry, a utility function provided by Django, which resides at django.utils.functional. It essential preloads an argument, so that when you call the resulting function later on, you don’t provide the already-loaded argument, but it gets sent to the original function anyway. I’ll probably write it up properly later, but hopefully the following code should give the general idea.

from django.utils.functional import curry

class ModelPlugins(object):
    def __init__(self, mount):
        self.mount = mount

    def __get__(self, instance, owner):
        return [curry(p, instance) for p in self.mount.plugins]

Now each item being returned is a callable, so the validation code can add arguments to it.

class Article(models.Model):
    validators = ModelPlugins(ValidationProvider)

    # ... fields and methods

    def validate(self, verbosity=1):
        for validator in self.validators:
            validator(verbosity).validate()

Internally, curried functions use the argument pass-through technique described above, so there’s no need include that logic here. As long as all the plugins accept two arguments — a model instance and a verbosity setting, everything will work just fine.

Uses

As described above, this plugin system could be used to provide a ilst of actions that may be performed, depending on what applications are loaded. From a Django perspective, this could be incredibly valuable for newforms-admin, as an easy way to add new actions for models or objects. (I, for one, would love a way to provide a “Settings” link for use with dbsettings).

In addition, I would very much like to see it used for certain types of validation, as shown above. Jacob‘s working on model validation at the moment, but I expect that’ll only be useful on models you control. In particular, I’m thinking about Django’s built-in User model, which nearly every Django project uses. Logging users in is case-sensitive, but while there’s a way to log users in without case-sensitivity, it still doesn’t prevent two users registering nearly-identical usernames that are only different in what case they use. Also, users often request a way to make sure that email addresses are unique, which isn’t enforced by default.

If Django provided a mount point for validation plugins that would be used whenever a new user is created, individual projects could implement whatever they need, without a problem. Of course, I don’t know how James would feel about this, though, as many of the features provided by django-registration could then be reduced to plugins distributed on djangosnippets.

Future development

6 lines of code. Hardly enough to consider releasing as an application, even though it performs the features of a basic framework. I’ve released it into the wild, in hopes that it’ll be useful other projects. If you’d like to use it, just drop it in an app that provides pluggability and go from there. If you’d like to add more features to it, go right ahead. I’d be interested to hear what you’re doing with it, but I’m considering it public domain, given that it’s so darn simple. If you find enough need for new features to make it a full-fledged app, feel free, but remember: KISS.

Other notes

If you’ve worked much with signals, you might notice some similarities. In effect, the overall process is quite similar to how signals work. After all, both are designed to encourage loose coupling, so it’s no surprise that there are certain similarities. The underlying implementation is quite different, however. I started out using a global dictionary to maintain a list of registered mount points and map them to lists of registered plugins, which would have been somewhat similar to how PyDispatcher works at the moment.

But by moving the registration and maintenance onto the mount point class itself, it becomes much smaller and easier to use and maintain. Interestingly, there’s been some concern about the speed of signals, and Brian Harring has proposed a refactoring using this same basic technique. I actually owe much of this design to his emails on the subject.

Conclusion

Essentially, my intent was to provide the absolute bare-bones minimum necessary to achieve the desired effect. There may be useful features to add, there may be security or threading issues to address, and whoever finds those can do what’s necessary and report back to the community. I just hope that by starting from such minimal code, we can avoid any assumptions about needing Zope interfaces just to encourage pluggable code.