This tutorial will teach you how to create and use Python generators effectively. You’ll learn the basics, explore advanced use cases, and solve real-world problems using generators.
Step 1: What Are Python Generators?
Generators are a special type of iterator in Python that allow you to iterate over a sequence of values without storing them in memory. They are defined using the yield keyword instead of return. Generators are memory-efficient and ideal for working with large datasets or infinite sequences.
def simple_generator():
yield 1
yield 2
yield 3
for value in simple_generator():
print(value)Practice Exercise
Create a generator function count_up_to(n) that yields numbers from 1 to n. Test it by printing numbers up to 5.
Show Solution
def count_up_to(n):
num = 1
while num <= n:
yield num
num += 1
for number in count_up_to(5):
print(number)Step 2: How Generators Work Under the Hood
Generators maintain their state between iterations. When a generator function is called, it returns a generator object but doesn't start executing the function. Execution begins when you call next() or iterate over the generator. The function pauses at each yield statement and resumes from there on the next iteration.
def stateful_generator():
print('Start')
yield 'First'
print('Resume')
yield 'Second'
gen = stateful_generator()
print(next(gen))
print(next(gen))Practice Exercise
Write a generator function alternate_letters(word) that yields alternating letters of a given word. For example, for 'python', it should yield 'p', 't', 'o'.
Show Solution
def alternate_letters(word):
for i in range(0, len(word), 2):
yield word[i]
for letter in alternate_letters('python'):
print(letter)Step 3: Using Generators for Infinite Sequences
Generators are perfect for creating infinite sequences because they don't store all values in memory. For example, you can create a generator that produces an infinite sequence of numbers or Fibonacci numbers.
def infinite_sequence():
num = 0
while True:
yield num
num += 1
# Use with caution: this will run forever!
# for num in infinite_sequence():
# print(num)Practice Exercise
Create a generator function fibonacci() that yields the Fibonacci sequence indefinitely. Use it to print the first 10 Fibonacci numbers.
Show Solution
def fibonacci():
a, b = 0, 1
while True:
yield a
a, b = b, a + b
fib_gen = fibonacci()
for _ in range(10):
print(next(fib_gen))Step 4: Generator Expressions
Generator expressions are a concise way to create generators. They are similar to list comprehensions but use parentheses instead of square brackets. They are memory-efficient because they generate items on the fly.
squares = (x * x for x in range(10))
for square in squares:
print(square)Practice Exercise
Use a generator expression to create a generator that yields the cubes of numbers from 1 to 10. Print the cubes.
Show Solution
cubes = (x ** 3 for x in range(1, 11))
for cube in cubes:
print(cube)Step 5: Chaining Generators
You can chain multiple generators together using the yield from syntax. This allows you to combine the output of multiple generators into a single generator.
def generator1():
yield 'A'
yield 'B'
def generator2():
yield 'C'
yield 'D'
def combined_generator():
yield from generator1()
yield from generator2()
for value in combined_generator():
print(value)Practice Exercise
Create two generators: one that yields even numbers from 0 to 10 and another that yields odd numbers from 1 to 9. Combine them into a single generator and print the combined sequence.
Show Solution
def even_numbers():
for i in range(0, 11, 2):
yield i
def odd_numbers():
for i in range(1, 10, 2):
yield i
def combined():
yield from even_numbers()
yield from odd_numbers()
for num in combined():
print(num)Step 6: Real-World Use Case: Processing Large Files
Generators are ideal for processing large files line by line without loading the entire file into memory. This is useful for log files, CSV files, or any large dataset.
def read_large_file(file_path):
with open(file_path, 'r') as file:
for line in file:
yield line
for line in read_large_file('large_log.txt'):
print(line.strip())Practice Exercise
Write a generator function filter_lines(file_path, keyword) that reads a file line by line and yields only lines containing a specific keyword. Test it with a sample file.
Show Solution
def filter_lines(file_path, keyword):
with open(file_path, 'r') as file:
for line in file:
if keyword in line:
yield line
for line in filter_lines('sample.txt', 'error'):
print(line.strip())Step 7: Advanced: Sending Data to Generators
Generators can receive data using the send() method. This allows two-way communication between the caller and the generator. The generator can yield a value and then receive a value to continue execution.
def interactive_generator():
while True:
received = yield
print(f'Received: {received}')
gen = interactive_generator()
next(gen) # Start the generator
gen.send('Hello')
gen.send('World')Practice Exercise
Create a generator function accumulator() that accumulates values sent to it using send(). It should yield the current total each time a value is sent.
Show Solution
def accumulator():
total = 0
while True:
value = yield total
if value is not None:
total += value
acc = accumulator()
next(acc) # Start the generator
print(acc.send(10)) # Output: 10
print(acc.send(20)) # Output: 30Step 8: Advanced: Generator Pipelines
You can create pipelines by chaining multiple generators together. Each generator in the pipeline processes data and passes it to the next generator. This is useful for data transformation tasks.
def square_numbers(nums):
for num in nums:
yield num * num
def filter_even(nums):
for num in nums:
if num % 2 == 0:
yield num
numbers = range(10)
pipeline = filter_even(square_numbers(numbers))
for num in pipeline:
print(num)Practice Exercise
Create a pipeline with three generators: one that squares numbers, one that filters even numbers, and one that adds 10 to each number. Test it with numbers from 1 to 10.
Show Solution
def square_numbers(nums):
for num in nums:
yield num * num
def filter_even(nums):
for num in nums:
if num % 2 == 0:
yield num
def add_ten(nums):
for num in nums:
yield num + 10
numbers = range(1, 11)
pipeline = add_ten(filter_even(square_numbers(numbers)))
for num in pipeline:
print(num)Sign in to take Cornell notes on this lesson — they save automatically and stay with your account.