Python Generators

Generators in Python

There is more work in creating an iterator in Python. We need to implement a class with __iter__() and __next__() method, keep track of internal states, and raise the StopIteration exception when there is no more value to be returned.

This is a little bit more complicated to create iterators. This is where generators come to help.

Python generators are an easy way of building iterators. When using generators, all the work to create iterators is handled automatically.

A generator is a function that returns an object (iterator) which we can iterate over. The values from the generator object are fetched one value at a time.

Create Generators in Python

Python makes it simple to create a generator. We need to define a normal function with a yield statement instead of a return statement.

If a function includes at least one yield statement (it can contain other yield or return statements), it becomes a generator function. Both yield and return will return some value from a function.

The difference between a return or a yield in a function is that while a return statement terminates a function, a yield statement pauses the function saving all its states and make it possible to continue from there on the next call.

Differences between Normal function and Generator function

Here are some points that can clear the difference between Generator Function and Normal function.

• Normal function has one return statement, whereas generator function can use one or more yield statements.
• When a generator function is called, it returns an object (iterator) but does not start execution immediately.
• __iter__() and __next__() methods are implemented automatically inside generator functions. So we can iterate through items using next().
• Once the generator function yields, the function is paused, and the control is transferred to the caller.
• Generator function remembers local variables and their states between successive calls.
• When the generator function terminates, StopIteration is raised automatically on further calls.

In the following example, we will illustrate all of the points stated above. We will create a generator function named my_generator() with several yield statements.

def my_generator():
    n = 0
    
    print("This is printed first")
    yield n

    n += 1
    print("This is printed second")
    yield n

    n += 1
    print("This is printed at last")
    yield n

Now we can execute the code below.

# It returns an object without starting execution immediately. 
x = my_generator()

# we can iterate through items using next()
print(next(x))

# the function is paused when the function yields and the control is transferred to the caller.
print(next(x))

print(next(x))

# Finally, when the function terminates. StopIteration is raised automatically on further calls.
print(next(x))

The output of the above code can be given as follows:

This is printed first
0
This is printed second
1
This is printed at last
2

Traceback (most recent call last)
..
---> 28 next(x)
...
StopIteration: 

As we can see above, the variable n value is remembered between each call.

Generator functions do not destroy local variables when the function yields, unlike normal functions.

The generator object can be iterated only one. We need to create another object by typing x = my_generator() to restart the process.

Generators can be used with for loops directly.

We can use a for loop with generators directly because it can take an iterator and iterates over it using the next() function. It automatically ends when StopIteration is raised.

def my_generator():
    n = 0
    
    print("This is printed first")
    yield n

    n += 1
    print("This is printed second")
    yield n

    n += 1
    print("This is printed at last")
    yield n

# Using for loop
for item in my_generator():
    print(item)

Output

This is printed first
0
This is printed second
1
This is printed at last
2    

Python Generators with a Loop

Usually, generator functions are implemented with a loop having a suitable terminating condition.

In the following example, we will define a generator function that reverses a string.

def reverse_str(my_str):
    length = len(my_str)
    for i in range(length - 1, -1, -1):
        yield my_str[i]

for c in reverse_str("apple"):
    print(c)

Output

e
l
p
p
a

Above, we used the range() function to get the index reverse order using the for loop.

Note: This generator function works with strings and also with other kinds of iterables like list, tuple, set, etc.

Python Generator Expression

Using generator expressions makes it easier to create simple generators.

Similar to the lambda functions that create anonymous functions, generator expressions create anonymous generator functions.

The syntax for generator expression is similar to the syntax of a list comprehension. The syntax difference is that the generator expression uses round parentheses () while list comprehension uses square brackets [].

The difference between a generator expression and a list comprehension is that a generator expression returns one item on each iteration while a list comprehension returns the entire list.

The generator expression has lazy execution (returning one item when asked). This is why generator expression is more memory efficient than an equivalent list comprehension.

my_list = [0, 2 , 4, 6, 8]

# add one to each number using list comprehension
transformed_list = [x+1 for x in my_list ]

# using a generator expression can give the same result
# generator expressions use parenthesis ()
gen = (x+1 for x in my_list) 

print(transformed_list )
print(gen)

Output

[1, 3, 5, 7, 9] 
<generator object <genexpr> at 0x7fdc01a09550>

As we can see above, the generator expression did not return the result immediately. Instead, it returned a generator object that produces items in a lazy way (on demand).

In the following example, we will start to get items from the generator.

my_list = [0, 2 , 4, 6, 8]

gen = (x+1 for x in my_list) 

print(next(gen))

print(next(gen))

print(next(gen))

print(next(gen))

print(next(gen))

The above program will produce the following output:

1
3
5
7
9

Traceback (most recent call last)
...
---> 15 print(next(gen))
...
StopIteration:

We can also use generator expressions as functions arguments. And this case, we can use them without round parentheses.

>>> max(x+1 for x in my_list)
9
>>> sum(x+1 for x in my_list)
25

Why use Python Generators?

There are different reasons to make us use generators.

Generators are easy to implement

Generators are easier to implement in a clear and compact way as compared to their iterator class. In the following example, we will implement a sequence of power of 3 using an iterator class.

class PowThree:
    def __init__(self, max=0):
        self.a = 0
        self.max = max

    def __iter__(self):
        return self

def __next__(self):
    if self.a > self.max:
        raise StopIteration

    result = 3 ** self.a
    selg.a += 1
    return result

The above program is lengthy and complicated. Now, we will do the same with the help of the generator function.

def PowThreeGen(max=0):
    a = 0
    while a < max:
        yield 3 ** a
        a +=  1

As we can see above, the implementation of the generator is much cleaner and compact.

Generators are memory-efficient

When using a normal function to return a sequence, it creates the entire sequence in memory before returning the result. This consumes so much memory, especially if the number of items in the sequence is large.

Generator implementation of large sequences is memory-friendly and recommended since it only returns one item at a time.

Generators can represent Infinite Stream

Generators are a great way to represent an infinite stream of data. As we know, infinite streams cannot be stored in memory. This is why we use generators that produce only one item at a time to represent infinite stream data.

In the following example, we will define a generator function that produces all the odd numbers.

def all_odd():
    n = 1
    while True:
        yield n
        n += 1

Generators can be used with pipeline

We can use multiple generators by pipelining a series of operations.

Let us suppose that we have a generator that produces odd numbers and another generator for squaring numbers.

If we want to find out the sum of squares of numbers produced by the odd generator, we can do it by pipelining the output of generator functions together.

def odd_num(max):
    n = 1 
    for i in range(0, max):
        yield n
        n += 2

def square(nums):
    for num in nums:
        yield num**2

print(sum(square(odd_num(11))))

Output

1771

This way of combining multiple generators using pipelining is efficient and easy to read.