This is an introduction to dunder methods in Python, to help you understand what they are and what they are for.

(If you are new here and have no idea what a Pydon't is, you may want to read the Pydon't Manifesto.)

Introduction

Python is a language that has a rich set of built-in functions and operators that work really well with the built-in types. For example, the operator + works on numbers, as addition, but it also works on strings, lists, and tuples, as concatenation:

>>> 1 + 2.3
3.3
>>> [1, 2, 3] + [4, 5, 6]
[1, 2, 3, 4, 5, 6]

But what is it that defines that + is addition for numbers (integers and floats) and concatenation for lists, tuples, strings? What if I wanted + to work on other types? Can I do that?

The short answer is “yes”, and that happens through dunder methods, the object of study in this Pydon't. In this Pydon't, you will

• understand what are dunder methods;
• why they are called like that;
• see various useful dunder methods;
• learn about what dunder methods correspond to what built-ins;
• write your own dunder methods for example classes; and
• realise that dunder methods are like any other method you have written before.

What are dunder methods?

In Python, dunder methods are methods that allow instances of a class to interact with the built-in functions and operators of the language. The word “dunder” comes from “double underscore”, because the names of dunder methods start and end with two underscores, for example __str__ or __add__. Typically, dunder methods are not invoked directly by the programmer, making it look like they are called by magic. That is why dunder methods are also referred to as “magic methods” sometimes.¹

Dunder methods are not called magically, though. They are just called implicitly by the language, at specific times that are well-defined, and that depend on the dunder method in question.

The dunder method everyone knows

If you have defined classes in Python, you are bound to have crossed paths with a dunder method: __init__. The dunder method __init__ is responsible for initialising your instance of the class, which is why it is in there that you usually set a bunch of attributes related to arguments the class received.

For example, if you were creating an instance of a class Square, you would create the attribute for the side length in __init__:

class Square:
    def __init__(self, side_length):
        """__init__ is the dunder method that INITialises the instance.

        To create a square, we need to know the length of its side,
        so that will be passed as an argument later, e.g. with Square(1).
        To make sure the instance knows its own side length,
        we save it with self.side_length = side_length.
        """
        print("Inside init!")
        self.side_length = side_length

sq = Square(1)
# Inside init!

If you run the code above, you will see the message “Inside init!” being printed, and yet, you did not call the method __init__ directly! The dunder method __init__ was called implicitly by the language when you created your instance of a square.

Why do dunder methods start and end with two underscores?

The two underscores in the beginning and end of the name of a dunder method do not have any special significance. In other words, the fact that the method name starts and ends with two underscores, in and of itself, does nothing special. The two underscores are there just to prevent name collision with other methods implemented by unsuspecting programmers.

Think of it this way: Python has a built-in called sum. You can define sum to be something else, but then you lose access to the built-in that sums things, right?

>>> sum(range(10))
45
>>> sum = 45
>>> sum(range(10))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable

Often, you see beginners using sum as a variable name because they do not know sum is actually a built-in function. If the built-in was named __sum__ instead of sum, it would be much more difficult for you to override it by mistake, right? But it would also make it much less convenient to use sum...

However, for magic methods, we do not need their names to be super convenient to type, because you almost never type the name of a magic method. Therefore, Python decided that the magic methods would have names that start and end with two underscores, to make it less likely that someone would override one of those methods by accident!

All in all, dunder methods are just like any other method you have implemented, with the small exception that dunder methods can be called implicitly by the language.

Operator overloading in Python and dunder methods

All Python operators, like +, ==, and in, rely on dunder methods to implement their behaviour.

For example, when Python encounters the code value in container, it actually turns that into a call to the appropriate dunder method __contains__, which means that Python actually runs the expression container.__contains__(value).

Let me show you:

>>> my_list = [2, 4, 6]

>>> 3 in my_list
False
>>> my_list.__contains__(3)
False

>>> 6 in my_list
True
>>> my_list.__contains__(6)
True

Therefore, when you want to overload certain operators to make them work in a custom way with your own objects, you need to implement the respective dunder methods.

So, if you were to create your own type of container, you could implement the dunder method __contains__ to make sure that your containers could be on the right-hand side of an expression with the operator in.

List of dunder methods and their interactions

As we have seen, dunder methods are (typically) called implicitly by the language... But when? The dunder method __init__ is called when initialising an instance of a class, but what about __str__, or __bool__, or other dunder methods?

The table that follows lists all dunder methods together with one or more (simplified) usage examples that would implicitly call the respective dunder method. This may include brief descriptions of situations where the relevant dunder method might be called, or example function calls that depend on that dunder method. These example situations may have caveats associated, so be sure to read the documentation on dunder methods whenever you want to play with a dunder method you are unfamiliar with.

The row order of the table matches the order in which these dunder methods are mentioned in the “Data Model” page of the documentation, which does not imply any dependency between the various dunder methods, nor does it imply a level of difficulty in understanding the methods.

Dunder method	Usage / Needed for	Learn more
__init__	Initialise object	docs
__new__	Create object	docs
__del__	Destroy object	docs
__repr__	Compute “official” string representation / repr(obj)	blog; docs
__str__	Pretty print object / str(obj) / print(obj)	blog; docs
__bytes__	bytes(obj)	docs
__format__	Custom string formatting	blog; docs
__lt__	obj < ...	docs
__le__	obj <= ...	docs
__eq__	obj == ...	docs
__ne__	obj != ...	docs
__gt__	obj > ...	docs
__ge__	obj >= ...	docs
__hash__	hash(obj) / object as dictionary key	docs
__bool__	bool(obj) / define Truthy/Falsy value of object	blog; docs
__getattr__	Fallback for attribute access	docs
__getattribute__	Implement attribute access: obj.name	docs
__setattr__	Set attribute values: obj.name = value	docs
__delattr__	Delete attribute: del obj.name	docs
__dir__	dir(obj)	docs
__get__	Attribute access in descriptor	docs
__set__	Set attribute in descriptor	docs
__delete__	Attribute deletion in descriptor	docs
__init_subclass__	Initialise subclass	docs
__set_name__	Owner class assignment callback	docs
__instancecheck__	isinstance(obj, ...)	docs
__subclasscheck__	issubclass(obj, ...)	docs
__class_getitem__	Emulate generic types	docs
__call__	Emulate callables / obj(*args, **kwargs)	docs
__len__	len(obj)	docs
__length_hint__	Estimate length for optimisation purposes	docs
__getitem__	Access obj[key]	blog; docs
__setitem__	obj[key] = ... or obj[]	blog; docs
__delitem__	del obj[key]	blog; docs
__missing__	Handle missing keys in dict subclasses	docs
__iter__	iter(obj) / for ... in obj (iterating over)	docs
__reversed__	reverse(obj)	docs
__contains__	... in obj (membership test)	docs
__add__	obj + ...	blog; docs
__radd__	... + obj	blog; docs
__iadd__	obj += ...	blog; docs
__sub__ 2 3	obj - ...	blog; docs
__mul__ 2 3	obj * ...	blog; docs
__matmul__ 2 3	obj @ ...	blog; docs
__truediv__ 2 3	obj / ...	blog; docs
__floordiv__ 2 3	obj // ...	blog; docs
__mod__ 2 3	obj % ...	blog; docs
__divmod__ 2	divmod(obj, ...)	blog; docs
__pow__ 2 3	obj ** ...	blog; docs
__lshift__ 2 3	obj << ...	blog; docs
__rshift__ 2 3	obj >> ...	blog; docs
__and__ 2 3	obj & ...	blog; docs
__xor__ 2 3	obj ^ ...	blog; docs
__or__ 2 3	obj | ...	blog; docs
__neg__	-obj (unary)	blog; docs
__pos__	+obj (unary)	blog; docs
__abs__	abs(obj)	blog; docs
__invert__	~obj (unary)	blog; docs
__complex__	complex(obj)	docs
__int__	int(obj)	docs
__float__	float(obj)	docs
__index__	Losslessly convert to integer	docs
__round__	round(obj)	docs
__trunc__	math.trunc(obj)	docs
__floor__	math.floor(obj)	docs
__ceil__	math.ceil(obj)	docs
__enter__	with obj (enter context manager)	docs
__exit__	with obj (exit context manager)	docs
__await__	Implement awaitable objects	docs
__aiter__	aiter(obj)	docs
__anext__	anext(obj)	docs
__aenter__	async with obj (enter async context manager)	docs
__aexit__	async with obj (exit async context manager)	docs

Exploring a dunder method

Whenever I learn about a new dunder method, the first thing I do is to play around with it.

Below, I share with you the three steps I follow when I'm exploring a new dunder method:

1. try to understand when the dunder method is called;
2. implement a stub for that method and trigger it with code; and
3. use the dunder method in a useful situation.

I will show you how I follow these steps with a practical example, the dunder method __missing__.

What is the dunder method for?

What is the dunder method __missing__ for? The documentation for the dunder method __missing__ reads:

“Called by dict.__getitem__() to implement self[key] for dict subclasses when key is not in the dictionary.”

In other words, the dunder method __missing__ is only relevant for subclasses of dict, and it is called whenever we cannot find a given key in the dictionary.

How to trigger the dunder method?

In what situations, that I can recreate, does the dunder method __missing__ get called?

From the documentation text, it looks like we might need a dictionary subclass, and then we need to access a key that does not exist in that dictionary. Thus, this should be enough to trigger the dunder method __missing__:

class DictSubclass(dict):
    def __missing__(self, key):
        print("Hello, world!")

my_dict = DictSubclass()
my_dict["this key isn't available"]
# Hello, world!

Notice how barebones the code above is: I just defined a method called __missing__ and made a print, just so I could check that __missing__ was being called.

Now I am going to make a couple more tests, just to make sure that __missing__ is really only called when trying to get the value of a key that doesn't exist:

class DictSubclass(dict):
    def __missing__(self, key):
        print(f"Missing {key = }")

my_dict = DictSubclass()
my_dict[0] = True
if my_dict[0]:
    print("Key 0 was `True`.")
# Prints: Key 0 was `True`
my_dict[1]  # Prints: Missing key = 1

Using the dunder method in a useful situation

Now that we have a clearer picture of when __missing__ comes into play, we can use it for something useful. For example, we can try implementing defaultdict based on __missing__.

defaultdict is a container from the module collections, and it's just like a dictionary, except that it uses a factory to generate default values when keys are missing.

For example, here is an instance of defaultdict that returns the value 0 by default:

from collections import defaultdict

olympic_medals = defaultdict(lambda: 0)  # Produce 0 by default
olympic_medals["Phelps"] = 28

print(olympic_medals["Phelps"])  # 28
print(olympic_medals["me"])  # 0

So, to reimplement defaultdict, we need to accept a factory function, we need to save that factory, and we need to use it inside __missing__.

Just as a side note, notice that defaultdict not only returns the default value, but also assigns it to the key that wasn't there before:

>>> from collections import defaultdict
>>> olympic_medals = defaultdict(lambda: 0)  # Produce 0 by default
>>> olympic_medals
defaultdict(<function <lambda> at 0x000001F15404F1F0>, {})
>>> # Notice the underlying dictionary is empty -------^^
>>> olympic_medals["me"]
0
>>> olympic_medals
defaultdict(<function <lambda> at 0x000001F15404F1F0>, {'me': 0})
>>> # It's not empty anymore --------------------------^^^^^^^^^

Given all of this, here is a possible reimplementation of defaultdict:

class my_defaultdict(dict):
    def __init__(self, default_factory, **kwargs):
        super().__init__(**kwargs)
        self.default_factory = default_factory

    def __missing__(self, key):
        """Populate the missing key and return its value."""
        self[key] = self.default_factory()
        return self[key]

olympic_medals = my_defaultdict(lambda: 0)  # Produce 0 by default
olympic_medals["Phelps"] = 28

print(olympic_medals["Phelps"])  # 28
print(olympic_medals["me"])  # 0

Conclusion

Here's the main takeaway of this Pydon't, for you, on a silver platter:

“Dunder methods are specific methods that allow you to specify how your objects interact with the Python syntax, its keywords, operators, and built-ins.”

This Pydon't showed you that:

• dunder methods are methods that are called implicitly by the Python language in specific situations;
• “dunder” comes from “double underscore”, referring to the two underscores that are the prefix and the suffix of all dunder methods;
• dunder methods are sometimes called magic methods because they are often called without explicit calls;
• learning about a new dunder method can be done through a series of small, simple steps; and
• dunder methods are regular Python methods of regular Python classes.

If you liked this Pydon't be sure to leave a reaction below and share this with your friends and fellow Pythonistas. Also, don't forget to subscribe to the newsletter so you don't miss a single Pydon't!

Become a better Python 🐍 developer, drop by drop 💧

Get a daily drop of Python knowledge. A short, effective tip to start writing better Python code: more idiomatic, more effective, more efficient, with fewer bugs. Subscribe here.

References

• Python 3 Documentation, Data model, https://docs.python.org/3/reference/datamodel.html [last accessed 09-07-2022];