If you’ve ever written a loop in Python, you’ve used an iterator without realizing it. If you’ve used range(), map(), or filter(), you’ve benefited from generators. These concepts are fundamental to Python, yet many developers don’t fully understand how they work or why they matter.
Iterators and generators enable lazy evaluation—computing values only when needed. This approach saves memory, improves performance, and enables elegant solutions to complex problems. This guide demystifies these concepts and shows you how to harness their power.
Why Iterators and Generators Matter
Consider this scenario: you need to process a file with one million lines. With a traditional approach, you’d load all lines into memory:
# ❌ Loads entire file into memory
with open('huge_file.txt') as f:
lines = f.readlines() # All million lines in memory!
for line in lines:
process(line)
With generators, you process one line at a time:
# ✓ Processes one line at a time
with open('huge_file.txt') as f:
for line in f: # Generator-based iteration
process(line)
The second approach uses a fraction of the memory. This is the power of iterators and generators.
Understanding Iterators
What is an Iterator?
An iterator is an object that implements two methods:
__iter__(): Returns the iterator object itself__next__(): Returns the next value from the sequence
When __next__() has no more values, it raises StopIteration.
The Iterator Protocol
Let’s create a simple iterator:
class CountUp:
"""Iterator that counts from 1 to max"""
def __init__(self, max):
self.max = max
self.current = 0
def __iter__(self):
return self
def __next__(self):
self.current += 1
if self.current > self.max:
raise StopIteration
return self.current
# Usage
counter = CountUp(3)
print(next(counter)) # Output: 1
print(next(counter)) # Output: 2
print(next(counter)) # Output: 3
# next(counter) # Would raise StopIteration
# Or use in a loop
counter = CountUp(3)
for num in counter:
print(num) # Output: 1, 2, 3
Built-in Iterators
Python’s built-in types like lists, tuples, and strings are iterable (they implement __iter__), but they’re not iterators themselves. When you iterate over them, Python calls iter() to get an iterator:
# Lists are iterable but not iterators
numbers = [1, 2, 3]
print(hasattr(numbers, '__iter__')) # True
print(hasattr(numbers, '__next__')) # False
# Get an iterator from a list
iterator = iter(numbers)
print(hasattr(iterator, '__iter__')) # True
print(hasattr(iterator, '__next__')) # True
print(next(iterator)) # Output: 1
print(next(iterator)) # Output: 2
print(next(iterator)) # Output: 3
Understanding Generators
What is a Generator?
A generator is a function that returns an iterator. Instead of using return, it uses yield to produce values one at a time. When you call a generator function, it doesn’t execute immediately—it returns a generator object.
The yield Keyword
The yield keyword is the key difference between generators and regular functions:
# Regular function: returns all values at once
def regular_function():
return 1
return 2
return 3
# Generator: yields values one at a time
def generator_function():
yield 1
yield 2
yield 3
# Calling a regular function executes it immediately
result = regular_function()
print(result) # Output: 1
# Calling a generator function returns a generator object
gen = generator_function()
print(gen) # Output: <generator object generator_function at 0x...>
# Get values from the generator
print(next(gen)) # Output: 1
print(next(gen)) # Output: 2
print(next(gen)) # Output: 3
How Generators Maintain State
This is where generators shine. When you call next(), the generator resumes from where it left off, maintaining its local state:
def countdown(n):
"""Generator that counts down from n"""
print(f"Starting countdown from {n}")
while n > 0:
yield n
n -= 1
print("Countdown complete!")
gen = countdown(3)
print(next(gen)) # Output: Starting countdown from 3, then 3
print(next(gen)) # Output: 2
print(next(gen)) # Output: 1
print(next(gen)) # Output: Countdown complete!, then StopIteration
Notice how the generator remembers the value of n between calls. This is state preservation—a powerful feature.
Practical Generator Examples
Example 1: Fibonacci Sequence
def fibonacci(limit):
"""Generate Fibonacci numbers up to limit"""
a, b = 0, 1
while a < limit:
yield a
a, b = b, a + b
# Usage
for num in fibonacci(100):
print(num, end=' ') # Output: 0 1 1 2 3 5 8 13 21 34 55 89
Example 2: Reading Large Files
def read_large_file(filepath, chunk_size=1024):
"""Read a large file in chunks"""
with open(filepath, 'rb') as f:
while True:
chunk = f.read(chunk_size)
if not chunk:
break
yield chunk
# Usage: Process file without loading it all into memory
for chunk in read_large_file('huge_file.bin'):
process_chunk(chunk)
Example 3: Infinite Sequences
def infinite_counter(start=0):
"""Generate infinite sequence of numbers"""
n = start
while True:
yield n
n += 1
# Usage: Take only what you need
counter = infinite_counter()
first_five = [next(counter) for _ in range(5)]
print(first_five) # Output: [0, 1, 2, 3, 4]
Generator Expressions
What Are Generator Expressions?
Generator expressions are a concise syntax for creating generators. They’re similar to list comprehensions but use parentheses instead of square brackets:
# List comprehension: creates entire list in memory
squares_list = [x ** 2 for x in range(10)]
# Generator expression: creates values on-demand
squares_gen = (x ** 2 for x in range(10))
print(type(squares_list)) # Output: <class 'list'>
print(type(squares_gen)) # Output: <class 'generator'>
# Both produce the same values, but generator uses less memory
print(list(squares_gen)) # Output: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
Generator Expression Syntax
# Basic syntax
(expression for item in iterable)
# With condition
(expression for item in iterable if condition)
# With nested loops
(x * y for x in range(3) for y in range(3))
# Practical examples
numbers = [1, 2, 3, 4, 5]
# Filter even numbers
evens = (x for x in numbers if x % 2 == 0)
print(list(evens)) # Output: [2, 4]
# Transform and filter
doubled_evens = (x * 2 for x in numbers if x % 2 == 0)
print(list(doubled_evens)) # Output: [4, 8]
When to Use Generator Expressions
Generator expressions are perfect for one-time iterations:
# ✓ Good: One-time use
total = sum(x ** 2 for x in range(1000000))
# ✓ Good: Passing to functions
result = list(filter(lambda x: x > 5, (x ** 2 for x in range(100))))
# ❌ Not ideal: Multiple iterations
gen = (x ** 2 for x in range(10))
print(list(gen)) # Works
print(list(gen)) # Empty! Generator exhausted
Performance Comparison: Lists vs Generators
Memory Usage
import sys
# List: stores all values in memory
list_comp = [x ** 2 for x in range(1000)]
print(f"List size: {sys.getsizeof(list_comp)} bytes")
# Generator: stores only the current value
gen_exp = (x ** 2 for x in range(1000))
print(f"Generator size: {sys.getsizeof(gen_exp)} bytes")
# Output:
# List size: 9016 bytes
# Generator size: 128 bytes
Execution Time
import time
# List: creates all values upfront
start = time.time()
list_comp = [x ** 2 for x in range(1000000)]
list_time = time.time() - start
# Generator: creates values on-demand
start = time.time()
gen_exp = (x ** 2 for x in range(1000000))
gen_time = time.time() - start
print(f"List creation: {list_time:.6f} seconds")
print(f"Generator creation: {gen_time:.6f} seconds")
# Output:
# List creation: 0.050000 seconds
# Generator creation: 0.000001 seconds
Generators are dramatically faster because they don’t compute values until needed.
Practical Use Cases
Use Case 1: Processing Streams of Data
def process_log_file(filepath):
"""Process log file line by line"""
with open(filepath) as f:
for line in f:
if 'ERROR' in line:
yield line.strip()
# Usage
for error_line in process_log_file('app.log'):
alert(error_line)
Use Case 2: Generating Test Data
def generate_test_users(count):
"""Generate test user data"""
for i in range(count):
yield {
'id': i,
'name': f'User{i}',
'email': f'user{i}@example.com'
}
# Usage: Generate only what you need
for user in generate_test_users(1000000):
save_to_database(user)
Use Case 3: Pipeline Processing
def read_data(source):
"""Read data from source"""
for item in source:
yield item
def filter_data(data, predicate):
"""Filter data based on predicate"""
for item in data:
if predicate(item):
yield item
def transform_data(data, transformer):
"""Transform data"""
for item in data:
yield transformer(item)
# Usage: Chain generators for elegant pipelines
data = read_data(range(100))
filtered = filter_data(data, lambda x: x % 2 == 0)
transformed = transform_data(filtered, lambda x: x ** 2)
for result in transformed:
print(result, end=' ')
Best Practices
Best Practice 1: Use Generators for Large Datasets
# ❌ Inefficient: Loads entire dataset
def get_all_users():
return [user for user in database.query_all()]
# ✓ Efficient: Yields users one at a time
def get_all_users():
for user in database.query_all():
yield user
Best Practice 2: Use Generator Expressions for Simple Cases
# ✓ Clear and concise
total = sum(x ** 2 for x in numbers)
# ❌ Unnecessarily verbose
def square_generator(numbers):
for x in numbers:
yield x ** 2
total = sum(square_generator(numbers))
Best Practice 3: Document Generator Behavior
def process_items(items):
"""
Process items one at a time.
Yields:
Processed item
Note:
This generator consumes the input iterable.
Each item is yielded only once.
"""
for item in items:
yield process(item)
Common Pitfalls
Pitfall 1: Exhausting Generators
# ❌ Generator exhausted after first use
gen = (x ** 2 for x in range(5))
print(list(gen)) # Output: [0, 1, 4, 9, 16]
print(list(gen)) # Output: [] - generator exhausted!
# ✓ Create new generator or convert to list
gen = (x ** 2 for x in range(5))
result = list(gen)
print(result) # Output: [0, 1, 4, 9, 16]
print(result) # Output: [0, 1, 4, 9, 16] - list can be reused
Pitfall 2: Forgetting yield
# ❌ Returns list, not a generator
def not_a_generator():
return [1, 2, 3]
# ✓ Actually a generator
def is_a_generator():
yield 1
yield 2
yield 3
Pitfall 3: Modifying External State
# ❌ Dangerous: Generator depends on external state
counter = 0
def bad_generator():
global counter
while counter < 5:
yield counter
counter += 1
# ✓ Better: Generator is self-contained
def good_generator():
counter = 0
while counter < 5:
yield counter
counter += 1
Conclusion
Iterators and generators are fundamental Python concepts that enable efficient, elegant code:
- Iterators implement
__iter__()and__next__()to provide sequential access - Generators use
yieldto create iterators with minimal code - Generator expressions provide concise syntax for simple generators
- Lazy evaluation saves memory and improves performance
- State preservation in generators enables complex patterns
Key takeaways:
- Use generators for large datasets or infinite sequences
- Use generator expressions for simple, one-time iterations
- Understand that generators are exhausted after iteration
- Leverage lazy evaluation for performance benefits
- Document generator behavior clearly
Start incorporating generators into your code today. You’ll find that many data processing tasks become more efficient and elegant. As you grow comfortable with these concepts, you’ll discover that generators enable powerful patterns like pipelines, streaming, and lazy evaluation that transform how you write Python.
The journey from loops to generators is gradual. Begin with simple generator expressions, then explore function-based generators. Before long, you’ll be writing code that’s both more efficient and more Pythonic.
Comments