Understanding generators

Generators are special functions that can be interrupted at specific locations in their code and can be resumed at a later time with their execution context being restored. Generators are an easy way to create an iterator.

An iterator is created by an iterable object. An iterable is a class that implements the __iter__ method. When called, the __iter__ method must return an iterator object; that is, an object that implements the next method. In many cases, the implementations of the iterable and the iterator are in the same class, and the implementation of __iter__ just return self. The next method must return the next value of the iterator each time it is being called. With generators, all this is much easier. The implementation of the iterable and the iterator is just a function. This function yields values until the iterator completes. So, generators ease the implementation of iterators.

A function is a generator if it contains a yield expression. In this case, the function has a special behavior. When it is called, its content is not executed as it is for functions. Instead, a generator object is returned. Let's see an example:

def double(x, end):
    v = x
    while True:
        v = v*2
        if v > end:
            break
        yield v
    return

The double function is a generator because it contains a yield expression. This generator returns a series of values that are the doubles of their preceding value. The generator stops when the current value is bigger than the value provided in the end parameter. Let's see what happens when we create an instance of this generator:

g = double(3, 30)
g
<generator object double at 0x10de6fb48>

The g variable is a reference to a generator object. From that point, no code in the double function has be executed yet. Execution will start when a value is requested:

next(g)
6

On this first call, the generator was executed until the yield statement. The returned value is the initialization value (which is, 3 in our case) multiplied by 2. Each time a new value is requested, the generator takes another step in its loop:

next(g)
12
next(g)
24
next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Two more values are returned by the generator. On the last call, an exception is returned. This is how a generator completes. The double function returned because the computed value was 24 * 2 = 48, which is bigger than 30. When a generator completes (and, more generally, when an iterator completes), a StopIteration exception is raised. So this exception must be caught by the caller for it to know when an iterator has completed.

Generators have an additional feature compared to iterators. They can receive values from the caller, so they are not limited to only sending values. Let's create another generator very similar to the previous one:

def double_from(x, end):
    v = x
    while True:
        v = v*2
        if v > end:
            break
        v = yield v
    return

The only difference is that the v value is set from the result of the yield expression. This is how a generator can receive a value from the caller of the generator. To provide this value, the caller must use the send method of the generator instead of the next function:

g = double_from(3, 30)
next(g)
6
g.send(4)
8
g.send(1)
2
g.send(9)
18