Decorators are an extremely powerful tool that allows easily adding additional features to existing functions or modifying their behavior without having to change the code of the functions themselves. Most Python programmers have definitely encountered decorators while reading code, and almost as many have also used them in their own code. Even the built-in functions include a few of them. Still, I feel that too many Python programmers lack a fundamental understanding of how decorators actually work. And it's easy to see why: the syntax is totally weird, reading the source code often just leads to more confusion, and the material explaining the concept is riddled with obscure terminology.

In this article we will take a look at how decorators are implemented as a Python programming language feature and go through a few examples of how they can be used to solve real world problems. Hopefully after reading it you will have one more tool in your box to make your life a little bit easier!

A gentle introduction to decorators

The decorator pattern is a design pattern in which the behavior of existing functions, classes or methods is modified or augmented using reusable pieces of code called decorators. Using decorators can be a very effective way to provide separation of concerns and to make your code more readable, testable and most importantly, beautiful. In many scenarios, they provide a clean and easily understandable way to reuse code that would be difficult otherwise.

For example, they can be used to add common side effects to otherwise pure functions while keeping both the original function and the side effects easily testable, either separately or together. Many popular Python frameworks such as Flask use decorators to indicate which user-defined functions should be called by the framework code. Also, the Python standard library has many widely used decorators such as @property, @staticmethod and @dataclass that can help you modify the default behavior of your classes or methods.

Note: the terms decorator and decorator pattern used in this text refer specifically to the decorator syntax in Python programming language. It differs slightly from the original concept described in Design Patterns: Elements of Reusable Object-Oriented Software.

Consider the following scenario:

You are working on a large-scale data processing application and start getting complaints from customers that sometimes the application produces results with errors. After some investigation, you are able to verify that the claims are correct and even consistently reproducible in the production environment. However, when you try to reproduce the issue locally or in various testing environments, everything works correctly.

By inspecting the source code and the reported errors in the results, you are able to narrow the problem down to a specific set of functions. But it's still entirely unclear which function is responsible for the incorrect results. The results depend not only on the input data; during the process, the application also retrieves large amounts of data from the database. Because the production database is full of sensitive customer data, making a copy of it or accessing it directly is not an option.

Adding more detailed logging to the application would be a great first step towards finding the cause of the bug. So every time one of the functions that are likely to cause the bug is executed, we will collect and log the following information:

• Which of the functions was executed
• When was the function executed
• Which arguments were passed to the function
• How long did it take to execute the function
• What was the result

Here is the code of the functions:

def important_func(a):
    return a + 1

def very_important_func(a, b):
    return a + b

def super_important_func(a, b, c):
    return a + b * c

Now it's time to write some code to satisfy our requirements! This is probably one of the simplest possible ways to do that:

import json
from datetime import datetime
from time import time

def important_func(a):
    t_0 = time()
    result = a + 1
    t_delta = time() - t_0
    stats = {
        'function': 'important_func',
        'called_at': datetime.fromtimestamp(t_0).isoformat(),
        'args': (a,),
        'execution_time_ms': t_delta * 1000,
        'result': result
    }
    print(json.dumps(stats, indent=4))
    return result

Now calling the function important_func prints out the desired information:

>>> important_func(5)
{
    "function": "important_func",
    "called_at": "2023-02-17T20:53:30.110048",
    "args": [
        5
    ],
    "execution_time_ms": 0.002384185791015625,
    "result": 6
}
6

Even though this is a perfectly valid solution to the problem, it doesn't look very nice. The core logic of the function (a + 1) is buried between lines of code that are performing logging, which is just a secondary task. Also, this solution is not very generalizable: adding the same logic to every important function would be a tedious and error-prone process. So let's use the most common tool for reusing logic: functions! We should probably create a function that collects and prints out the required information. Here is the implementation:

def print_stats(func_name, t_0, args, result):
    t_delta = time() - t_0
    stats = {
        'function': 'important_func',
        'called_at': datetime.fromtimestamp(t_0).isoformat(),
        'args': args,
        'execution_time_ms': t_delta * 1000,
        'result': result
    }
    print(json.dumps(stats, indent=4))

Let's add it to our function:

def important_func(a):
    t_0 = time()
    result = a + 1
    print_stats('important_func', t_0, (a,), result)
    return result

This solution is already a lot more readable than our first try, but there is still much room for improvement. To get the desired results, we still need multiple steps to modify each of the functions:

1. Record the time when the function execution started and store it in a variable (t_0)
2. Perform the core logic of the function and store the result in a variable (result)
3. Pass the following information to the print_stats function: the name of the function, the timestamp stored in t_0, a list of arguments the function was called with and the result stored in result
4. Return the result

If only we could figure a way to alter the behavior of these functions without modifying the functions themselves... At this point many experienced Python programmers would probably think: "Hey, this looks like a perfect use case for decorators!"

This is how a solution with decorators could look like:

@log_stats
def important_func(a):
    return a + 1

@log_stats
def very_important_func(a, b):
    return a + b

@log_stats
def super_important_func(a, b, c):
    return a + b * c

Now we can verify that calling the functions produce the correct output:

>>> important_func(5)
{
    "function": "important_func",
    "called_at": "2023-02-17T21:14:26.307986",
    "args": [
        5
    ],
    "execution_time_ms": 0.0057220458984375,
    "result": 6
}
6
>>> very_important_func(2, 3)
{
    "function": "very_important_func",
    "called_at": "2023-02-17T21:14:35.115774",
    "args": [
        2,
        3
    ],
    "execution_time_ms": 0.004291534423828125,
    "result": 5
}
5
>>> super_important_func(1, 2, 3)
{
    "function": "super_important_func",
    "called_at": "2023-02-17T21:14:43.500108",
    "args": [
        1,
        2,
        3
    ],
    "execution_time_ms": 0.0050067901611328125,
    "result": 7
}
7

Everything looks completely right! Even though the functions have different names and even different number of parameters, the decorator was somehow able to correctly record and include them in the output. How is this possible? Decorators can feel very intimidating and even magical to programmers who are not familiar with them. The next sections will clarify what actually happens when you type the @ symbol in your code and how you can create your own decorators.

Note: Metaprogramming is a programming technique in which computer programs are able to modify themselves or create new programs while running. Python is such a highly dynamic interpreted language that the traditional line between program code and data becomes very blurry. Whether using Python decorators should be called metaprogramming or not depends on how you interpret the vague definition of metaprogramming.

Higher-order functions and closures

To understand how decorators work, we need to learn about a concept called higher-order functions first. A higher-order function is any function that takes one or more functions as arguments or returns a function.

Let's start with the simplest case: a function that takes another function as an argument:

def square(x):
    return x ** 2

def apply(func, value):
    return func(value)

First, we defined a regular function square that returns the square of a number. Then we defined another function apply that applies a function to a value. Because apply takes a function as an argument, it is a higher-order function.

Calling a higher-order function like this is no different than calling any other function. To pass a function as an argument, you can use its name, just like any other variable:

>>> apply(square, 3)
9

It is also possible to use built-in functions like print as arguments for our higher order function:

>>> apply(print, 'Hello world!')
Hello world!

Defining a higher-order function that returns a function is slightly more complicated. Let's create a function that returns another function that raises a number to the n:th power:

def nth_power(n):
    def nth_power_func(x):
        return x ** n
    return nth_power_func

When called, the nth_power function creates and returns a new function. The behavior of the returned function depends on the value of the argument n: calling the nth_power function with n = 2 produces a new function that raises a number always to the second power. The number to be raised is determined by the value of the argument x when the newly created function is called. Note how the inner function nth_power_func can access the variable n even after the execution of the outer function nth_power has been completed. This structure is called a closure: when the inner function is returned, the variables in the scope of the outer function are captured and stored alongside with the returned function.

We can easily verify that calling the nth_power function does indeed return another function instead of a number:

>>> nth_power(2)
<function nth_power.<locals>.nth_power_func at 0x7f82787128c0>

We can now create new functions by calling our higher-order function, assign them to variables, and then call them just like any other function:

>>> square = nth_power(2)
>>> cube = nth_power(3)
>>> square(3)
9
>>> cube(3)
27

Of course it is also possible to call the returned function without assigning it to a variable first. Although this can easily become confusing, especially to readers less familiar with the functional programming paradigm:

>>> nth_power(2)(3)
9

Let's create a higher-order function that both takes a function as an argument and returns a function next:

def plus_one(original_func):
    def func_plus_one(value):
        return original_func(value) + 1
    return func_plus_one

The function plus_one takes a function original_func as an argument and returns a new function that applies the function original_func to a value and adds one to the result. Let's use the square function as an example:

>>> square_plus_one = plus_one(square)
>>> square_plus_one(3)
10

We have now successfully modified the behavior of an existing function! The function square is decorated by the plus_one function.

Note: a programming language is is said to have first-class functions if functions can be treated like any other variables. This means that it is possible to pass functions as arguments to other functions and return them from functions. As demonstrated above, Python is clearly a programming language with first-class functions.

Writing your first decorator

Now that we know how to create and utilize higher-order functions, let's create and apply a decorator to a function using the decorator syntax. Here's a decorator that prints the result of a function:

def print_output(original_func):
    def decorated_func(value):
        result = original_func(value)
        print(f'The result is {result}')
        return result
    return decorated_func

Let's define a function and decorate it with the print_output decorator:

@print_output
def square(x):
    return x ** 2

Now when we call the function square, the result will be printed:

>>> square(3)
The result is 9
9

As you can see, decorators in Python are just syntactic sugar, an alternative way to define a function and pass it as an argument to another function. This:

@print_output
def square(x):
    return x ** 2

produces exactly the same outcome as this:

def square(x):
    return x ** 2

square = print_output(square)

Making decorators more generic

There is still one huge problem with our print_output decorator: it expects the decorated function to always have exactly one parameter. Let's define some functions with more than one parameter or zero parameters and try to decorate them with it:

@print_output
def sum_of_two(a, b):
    return a + b

@print_output
def zero():
    return 0

The code above does execute without any errors. Looks like it's be possible to define and decorate functions with the print_output decorator even though the number of parameters does not match. However, when we try to call the decorated functions, the interpreter starts throwing errors at us indicating that the functions were not called with the correct amount of arguments:

>>> sum_of_two(2, 3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: decorated_func() takes 1 positional argument but 2 were given
>>> zero()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: decorated_func() missing 1 required positional argument: 'value'

Let's fix the issue by making the decorator more generic so that it can accept more than one type of function. In Python, functions can have two types of arguments: positional and keyword-only. The * operator can be used in the parameters list to indicate that the function accepts any number of positional arguments. The ** operator is used similarly to allow any number of keyword-only arguments to be passed to the function. By convention, the names args and kwargs are usually used for this purpose. Inside the scope of the function, positional arguments are accessible through the parameter name, packed in a tuple. Keyword-only arguments are packed in a dictionary.

Here is an improved version of our print_output decorator that accepts any number of arguments:

def print_output(original_func):
    def decorated_func(*args, **kwargs):
        result = original_func(*args, **kwargs)
        print(f'The result is {result}')
        return result
    return decorated_func

Note how the original function is called using *args and **kwargs. This is called argument unpacking. This:

args = (1, 2, 3)
kwargs = {'foo': 'bar'}
func(*args, **kwargs)

Produces the same results as this:

func(1, 2, 3, foo='bar')

Let's try decorating the functions sum_of_two and zero again, now with the improved, more generic decorator:

@print_output
def sum_of_two(a, b):
    return a + b

@print_output
def zero():
    return 0

The functions can now be called with the correct number of arguments without errors:

>>> sum_of_two(2, 3)
The result is 5
5
>>> zero()
The result is 0
0

We have now successfully created a generic function decorator that can be used to decorate any function!

Note: parameters and arguments, what's the difference? Even though these two terms are often used interchangeably, they do not mean exactly the same thing. Parameters, also known as formal parameters, are the variables in a function definition. They act as placeholders for values that will be passed to the function. Arguments are the actual values that get passed to the function when it's executed.

Solving an actual problem with decorators

It's time to put all the knowledge together and finally solve an actual problem with decorators. Here is the implementation of the log_stats decorator that was used to solve the statistics logging problem introduced earlier in the article:

from time import time
from datetime import datetime
import json

def log_stats(original_func):
    def decorated_func(*args, **kwargs):
        t_0 = time()
        result = original_func(*args, **kwargs)
        t_delta = time() - t_0
        stats = {
            'function': original_func.__qualname__,
            'called_at': datetime.fromtimestamp(t_0).isoformat(),
            'args': args,
            'kwargs': kwargs,
            'execution_time_ms': t_delta * 1000,
            'result': result
        }
        print(json.dumps(stats, indent=4))
        return result
    return decorated_func

@log_stats
def important_func(a, b, c):
    return a + b + c

And a quick verification that it works as expected:

>>> important_func(1, 2, 3)
{
    "function": "important_func",
    "called_at": "2023-02-17T21:47:40.361943",
    "args": [
        1,
        2,
        3
    ],
    "kwargs": {},
    "execution_time_ms": 0.003814697265625,
    "result": 6
}
6

The implementation looks very similar to the print_output decorator in the previous section. Here are the main steps:

1. Store the current timestamp in a variable t_0
2. Execute the original function and store the result in a variable result
3. Get the current timestamp after the execution of the original function is completed, calculate the difference to the timestamp from step 1 and store it in a variable t_delta
4. Create a dictionary stats and store all the information we want to record there
5. Print the information from stats variable
6. Return the result stored in result variable

Note how the arguments of the decorated function are accessible through the variables args and kwargs. Because they are just regular variables, they can be used as values of the stats dictionary. The closure also captures the original function object and makes it accessible in the scope of the inner function through the variable original_func.

As we learned earlier, functions in Python are just a specific types of objects. When a new function object is created using the def keyword, some metadata, such as the name of the function, number of parameters, possible default parameter values and so on, get stored in the object's attributes. One of these attributes is the qualified name of the function. It can be accessed through the __qualname__ property of the function object.

Note: The main purpose of the inner function in the log_stats decorator is to call the original function. Functions like this are often called wrapper functions.

Decorators with parameters

The log_stats decorator is starting to look pretty useful but it's still missing some major features. Maybe the biggest problem is that it outputs the information using the print function. If you are working on an existing application it's very likely that there is already an established solution to logging, perhaps using the logging module from Python standard library, some third-party library or a custom solution.

The simplest way to make use of the existing logging solution in our logging decorator would be just replacing the print function call with the desired logging function or method call. This would be a perfectly viable solution if there was only one logger in the application. But what if the application has multiple loggers? Should we create one decorator for every logger? That doesn't sound right. The answer is, of course, to parametrize the decorator. Because decorators are just functions, it shouldn't be too difficult to add some additional parameters to them, right? The decorator should take a logging function as an argument and use it to output the statistics

Let's recap the decorator syntax. This decorator prints "Hello world!" every time the decorated function is called:

def hello_decorator(original_func):
    def decorated_func(*args, **kwargs):
        print('Hello world!')
        return original_func(*args, **kwargs)
    return decorated_func

And this is a function decorated with it:

@hello_decorator
def sum_of_two(a, b):
    return a + b

The decorator syntax above is equivalent to:

def sum_of_two(a, b):
    return a + b

sum_of_two = decorator(sum_of_two)

So placing the decorator statement @hello_decorator above the function definition sum_of_two performs the following steps:

1. Create a new function object by evaluating the sum_of_two function definition
2. Call the hello_decorator function using the function object created in step 1 as the only argument. The result is another function object
3. Assign the function object created in step 2 to the variable sum_of_two

It seems like if we want to use the decorator syntax, we can only use decorator functions that accept exactly one argument, which is the function to be decorated. This means adding additional parameters to decorator functions cannot be the solution to parametrizing decorators.

What we can do instead is create a function that accepts the arguments we need to parametrize the decorator and returns a new decorator function that only accepts a single argument. We can call such a function decorator creator:

def create_hello_decorator(message):
    def hello_decorator(original_func):
        def decorated_func(*args, **kwargs):
            print(message)
            return original_func(*args, **kwargs)
        return decorated_func
    return hello_decorator

decorator = create_hello_decorator('Hello to you too!')

@decorator
def sum_of_two(a, b):
    return a + b

>>> sum_of_two(1, 2)
Hello to you too!
3

Sure, the code above works but it still doesn't look very nice. Let's try making it a little bit nicer:

@create_hello_decorator('Hello again')
def sum_of_two(a, b):
    return a + b

Calling the decorated function now prints out the message that was passed as an argument to the decorator creator function:

>>> sum_of_two(1, 2)
Hello again
3

How does this work? Weren't decorators supposed to only take a single argument? Well, the decorator syntax also allows decorator statements to call functions that return decorators. So the code above is equivalent to:

def sum_of_two(a, b):
    return a + b

decorator = create_hello_decorator('Hello again')
sum_of_two = decorator(sum_of_two)

or written without the intermediate variable:

def sum_of_two(a, b):
    return a + b

sum_of_two = create_hello_decorator('Hello again')(sum_of_two)

Now that we understand how to write decorator creator functions, let's parametrize the log_stats decorator:

from time import time
from datetime import datetime
import json

def log_stats(logger_func):
    def decorator(original_func):
        def decorated_func(*args, **kwargs):
            t_0 = time()
            result = original_func(*args, **kwargs)
            t_delta = time() - t_0
            stats = {
                'function': original_func.__qualname__,
                'called_at': datetime.fromtimestamp(t_0).isoformat(),
                'args': args,
                'kwargs': kwargs,
                'execution_time_ms': t_delta * 1000,
                'result': result
            }
            logger_func(json.dumps(stats, indent=4))
            return result
        return decorated_func
    return decorator

Then we decorate two of our functions using a different logger for each:

from my_logger_module import logger_one, logger_two

@log_stats(logger_one.info)
def important_func(a, b, c):
    return a + b + c

@log_stats(logger_two.info)
def very_important_func(a, b):
    return a + b

Now we can verify that when we call the decorated functions, the statistics are logged using the correct logger:

>>> important_func(1, 2, 3)
INFO:logger_one:{
    "function": "important_func",
    "called_at": "2023-02-19T13:36:45.373267",
    "args": [
        1,
        2,
        3
    ],
    "kwargs": {},
    "execution_time_ms": 0.004291534423828125,
    "result": 6
}
6
>>> very_important_func(4, 5)
INFO:logger_two:{
    "function": "very_important_func",
    "called_at": "2023-02-19T13:36:52.197418",
    "args": [
        4,
        5
    ],
    "kwargs": {},
    "execution_time_ms": 0.003814697265625,
    "result": 9
}
9

All done! We now have a flexible decorator for adding statistics logging to any function. And it's relatively easy to include in any existing application as well, regardless of the logging solution used. A decorator like this can often prove to be extremely useful, especially when debugging applications running in environments to which we have limited access. Of course, this is just the first step in the debugging process, but with a solution like this you won't be left searching for a needle in a haystack the next time a user reports a problem.

Note: in the final revision of log_stats, the innermost function decorated_func has access to the variables logger_func and original_func from the scopes of both of its enclosing functions. Calling the log_stats function returns a closure that returns another, nested closure when called.

If you want to get a better understanding of your closures, you can use the __closure__ attribute of function objects to see which variables are captured.

Conclusion and what's next

I hope this little peek under the hood of Python's decorators has been helpful. Perhaps the decorator syntax feels a little less magical, and decorators in general make more sense now. Even though some programming concepts such as higher-order functions and closures can be tricky, there is no need to make them overly complicated: just remembering that Python functions are like any other objects that can be passed around is almost always enough.

Still, in this article, we have only scratched the surface of the possibilities decorators offer. In the next parts of this series, we will dive deeper into the topic and explore more ways you can make your life easier with them.