Late Binding Closures

Consider the following Python function designed to create a list of multiplier functions.

from typing import List, Callable
from rich.pretty import pprint
def create_multipliers_lambda() -> List[Callable[[int], int]]:
    return [lambda x : i * x for i in range(5)]

pprint(create_multipliers_lambda())

for multiplier in create_multipliers_lambda():
    print(multiplier(2))

[
│   <function create_multipliers_lambda.<locals>.<listcomp>.<lambda> at 0x1181f8e50>,
│   <function create_multipliers_lambda.<locals>.<listcomp>.<lambda> at 0x1181f8ee0>,
│   <function create_multipliers_lambda.<locals>.<listcomp>.<lambda> at 0x1181f8f70>,
│   <function create_multipliers_lambda.<locals>.<listcomp>.<lambda> at 0x118202040>,
│   <function create_multipliers_lambda.<locals>.<listcomp>.<lambda> at 0x1182020d0>
]

8
8
8
8
8

When invoking this function and using the multipliers:

One might expect the output to be:

0
2
4
6
8

Each lambda function multiplies its input x by a unique value of i from the range 0 to 4. However, the actual output is:

8
8
8
8
8

All multiplier functions return 8 because they all use the final value of i, which is 4, multiplied by 2.

Understanding Closures and Late Binding

Closures allow a nested function to capture variables from its enclosing scope. In Python, closures exhibit late binding, meaning that the values of variables used in closures are looked up at the time the inner function is called, not when it is defined.

In the provided example, the lambda functions capture the variable i from the enclosing scope. However, by the time any of these lambda functions are invoked, the loop has completed, and i holds its final value of 4. Consequently, all lambda functions reference this same i, resulting in each multiplier producing 4 * 2 = 8.

Why the Unexpected Output Occurs

The unexpected behavior stems from Python’s late binding behavior in closures. Here’s a step-by-step breakdown:

1. Loop Execution: The list comprehension [lambda x: i * x for i in range(5)] iterates over i from 0 to 4, creating five lambda functions.

2. Lambda Creation: Each lambda function is defined as lambda x: i * x. However, it doesn’t capture the current value of i at each iteration. Instead, it captures the variable i itself.

3. Closure Binding: All lambda functions share the same enclosing scope where i exists. They do not store the value of i at the time of their creation.

4. Function Invocation: When multiplier(2) is called for each lambda, it looks up the current value of i in the enclosing scope, which is 4 after the loop concludes.

5. Result: Each lambda effectively computes 4 * 2, resulting in 8 for all multipliers.

from typing import List, Callable

def create_multipliers_inner() -> List[Callable[[int], int]]:
    multipliers = []
    for i in range(5):
        def multiplier(x): # this is lambda x: i * x
            return i * x
        multipliers.append(multiplier)
    return multipliers

create_multipliers_inner()

[<function __main__.create_multipliers_inner.<locals>.multiplier(x)>,
 <function __main__.create_multipliers_inner.<locals>.multiplier(x)>,
 <function __main__.create_multipliers_inner.<locals>.multiplier(x)>,
 <function __main__.create_multipliers_inner.<locals>.multiplier(x)>,
 <function __main__.create_multipliers_inner.<locals>.multiplier(x)>]

Ruff will say:

Function definition does not bind loop variable i [B023]

for multiplier in create_multipliers_lambda():
    print(multiplier(2))

print("-" * 100)

for multiplier in create_multipliers_inner():
    print(multiplier(2))

8
8
8
8
8
----------------------------------------------------------------------------------------------------
i: 0
i: 1
i: 2
i: 3
i: 4
8
8
8
8
8

Workarounds

To achieve the expected behavior where each lambda function retains its own value of i, you can employ using default arguments in the lambda function.

By leveraging default arguments in the lambda function, you can capture the current value of i at each iteration:

def create_multipliers():
    return [lambda x, i=i : i * x for i in range(5)]

for multiplier in create_multipliers():
    print(multiplier(2))

0
2
4
6
8

Here, i=i sets the default value of i for each lambda at the time of its creation, effectively binding the current value of i to the lambda.

from functools import partial
from operator import mul

def create_multipliers():
    return [partial(mul, i) for i in range(5)]

for multiplier in create_multipliers():
    print(multiplier(2))

References And Further Readings

• https://docs.python-guide.org/writing/gotchas/#late-binding-closures