Python’s functools module is a treasure trove of utilities that make functional programming more elegant and efficient. While many developers know these functions exist, few understand how to use them effectively. Three functions in particular—partial, wraps, and lru_cache—solve common problems and deserve a place in every Python developer’s toolkit.
This guide explores these three functions in depth, showing you not just how they work, but when and why to use them in real projects.
Understanding the functools Module
The functools module provides higher-order functions and operations on callable objects. It’s part of Python’s standard library and focuses on functional programming patterns. The three functions we’ll explore address specific challenges:
partial: Create specialized functions from general oneswraps: Preserve function metadata when creating decoratorslru_cache: Cache function results for performance
Let’s dive into each one.
Part 1: functools.partial
What is partial?
functools.partial creates a new function by fixing some arguments of an existing function. It’s a way to create specialized versions of general functions without writing wrapper functions.
The Problem It Solves
Imagine you have a function that takes multiple arguments, but you frequently call it with the same values for some arguments:
# Without partial: repetitive code
def multiply(a, b):
return a * b
# You frequently need to multiply by 2
result1 = multiply(5, 2)
result2 = multiply(10, 2)
result3 = multiply(15, 2)
# Or you create a wrapper function
def double(x):
return multiply(x, 2)
result1 = double(5)
result2 = double(10)
result3 = double(15)
This is verbose and repetitive. partial solves this elegantly.
Basic Usage
from functools import partial
def multiply(a, b):
return a * b
# Create a specialized function using partial
double = partial(multiply, b=2)
# Now use it
print(double(5)) # Output: 10
print(double(10)) # Output: 20
print(double(15)) # Output: 30
Syntax and Parameters
partial(func, /, *args, **keywords)
func: The function to wrap*args: Positional arguments to fix**keywords: Keyword arguments to fix
Practical Example 1: API Client Configuration
from functools import partial
import requests
def make_request(method, url, base_url='https://api.example.com', timeout=10):
"""Make an HTTP request"""
full_url = f"{base_url}{url}"
response = requests.request(method, full_url, timeout=timeout)
return response.json()
# Create specialized functions for your API
get_request = partial(make_request, 'GET')
post_request = partial(make_request, 'POST')
# Create functions for a specific API endpoint
api_get = partial(get_request, base_url='https://api.example.com')
api_post = partial(post_request, base_url='https://api.example.com')
# Usage
# users = api_get('/users')
# new_user = api_post('/users', data={'name': 'Alice'})
Practical Example 2: Data Processing Pipeline
from functools import partial
def format_currency(value, currency='USD', decimal_places=2):
"""Format a number as currency"""
formatted = f"{value:,.{decimal_places}f}"
return f"{currency} {formatted}"
# Create specialized formatters
format_usd = partial(format_currency, currency='USD')
format_eur = partial(format_currency, currency='EUR')
format_crypto = partial(format_currency, currency='BTC', decimal_places=8)
# Usage
prices = [1000.5, 2500.75, 15000.123]
usd_prices = [format_usd(p) for p in prices]
print(usd_prices)
# Output: ['USD 1,000.50', 'USD 2,500.75', 'USD 15,000.12']
eur_prices = [format_eur(p) for p in prices]
print(eur_prices)
# Output: ['EUR 1,000.50', 'EUR 2,500.75', 'EUR 15,000.12']
Practical Example 3: Sorting with Custom Keys
from functools import partial
def compare_by_attribute(obj, attribute, reverse=False):
"""Extract attribute for comparison"""
return getattr(obj, attribute)
class Product:
def __init__(self, name, price, rating):
self.name = name
self.price = price
self.rating = rating
def __repr__(self):
return f"Product({self.name}, ${self.price}, {self.rating}★)"
products = [
Product('Laptop', 999, 4.5),
Product('Mouse', 25, 4.8),
Product('Monitor', 300, 4.2),
]
# Create specialized comparison functions
by_price = partial(compare_by_attribute, attribute='price')
by_rating = partial(compare_by_attribute, attribute='rating')
# Sort using partial functions
sorted_by_price = sorted(products, key=by_price)
sorted_by_rating = sorted(products, key=by_rating, reverse=True)
print("By price:", sorted_by_price)
print("By rating:", sorted_by_rating)
When to Use partial
- Creating specialized versions of general functions
- Configuring functions with default parameters
- Simplifying function signatures for callbacks
- Building function pipelines with fixed arguments
Part 2: functools.wraps
What is wraps?
functools.wraps is a decorator that preserves the metadata of the original function when you create a wrapper. It copies important attributes like __name__, __doc__, and __annotations__ from the wrapped function to the wrapper.
The Problem It Solves
When you create a decorator, the wrapper function replaces the original function’s metadata:
# Without wraps: metadata is lost
def timing_decorator(func):
def wrapper(*args, **kwargs):
import time
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print(f"Took {end - start:.4f} seconds")
return result
return wrapper
@timing_decorator
def calculate_sum(numbers):
"""Calculate the sum of numbers"""
return sum(numbers)
print(calculate_sum.__name__) # Output: wrapper (should be calculate_sum!)
print(calculate_sum.__doc__) # Output: None (should be the docstring!)
This breaks introspection and documentation tools. wraps solves this.
Basic Usage
from functools import wraps
import time
def timing_decorator(func):
@wraps(func) # Preserve metadata
def wrapper(*args, **kwargs):
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print(f"Took {end - start:.4f} seconds")
return result
return wrapper
@timing_decorator
def calculate_sum(numbers):
"""Calculate the sum of numbers"""
return sum(numbers)
print(calculate_sum.__name__) # Output: calculate_sum
print(calculate_sum.__doc__) # Output: Calculate the sum of numbers
What wraps Preserves
wraps copies these attributes:
__name__: Function name__doc__: Docstring__module__: Module name__qualname__: Qualified name__annotations__: Type hints__dict__: Function attributes
Practical Example 1: Validation Decorator
from functools import wraps
def validate_positive(*arg_names):
"""Decorator that validates arguments are positive"""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
# Validate keyword arguments
for name in arg_names:
if name in kwargs and kwargs[name] <= 0:
raise ValueError(f"{name} must be positive")
return func(*args, **kwargs)
return wrapper
return decorator
@validate_positive('width', 'height')
def calculate_area(width, height):
"""Calculate the area of a rectangle"""
return width * height
# Metadata is preserved
print(calculate_area.__name__) # Output: calculate_area
print(calculate_area.__doc__) # Output: Calculate the area of a rectangle
# Function works correctly
print(calculate_area(5, 10)) # Output: 50
print(calculate_area(-5, 10)) # Raises ValueError
Practical Example 2: Logging Decorator
from functools import wraps
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def log_calls(func):
"""Decorator that logs function calls"""
@wraps(func)
def wrapper(*args, **kwargs):
logger.info(f"Calling {func.__name__} with args={args}, kwargs={kwargs}")
try:
result = func(*args, **kwargs)
logger.info(f"{func.__name__} returned {result}")
return result
except Exception as e:
logger.error(f"{func.__name__} raised {type(e).__name__}: {e}")
raise
return wrapper
@log_calls
def divide(a, b):
"""Divide two numbers"""
return a / b
# Metadata is preserved
print(divide.__name__) # Output: divide
print(divide.__doc__) # Output: Divide two numbers
# Function works with logging
result = divide(10, 2) # Logs the call and result
Practical Example 3: Retry Decorator
from functools import wraps
import time
def retry(max_attempts=3, delay=1):
"""Decorator that retries a function on failure"""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(1, max_attempts + 1):
try:
return func(*args, **kwargs)
except Exception as e:
if attempt == max_attempts:
raise
print(f"Attempt {attempt} failed: {e}. Retrying in {delay}s...")
time.sleep(delay)
return wrapper
return decorator
@retry(max_attempts=3, delay=1)
def unstable_api_call():
"""Call an unstable API"""
import random
if random.random() < 0.7:
raise ConnectionError("API temporarily unavailable")
return "Success!"
# Metadata is preserved
print(unstable_api_call.__name__) # Output: unstable_api_call
print(unstable_api_call.__doc__) # Output: Call an unstable API
# Function retries on failure
# result = unstable_api_call()
When to Use wraps
- Creating any decorator that wraps a function
- Preserving function metadata for documentation
- Maintaining introspection capabilities
- Building decorator libraries
- Ensuring debugging tools work correctly
Part 3: functools.lru_cache
What is lru_cache?
functools.lru_cache is a decorator that caches function results based on arguments. LRU stands for “Least Recently Used”—when the cache is full, the least recently used item is discarded. It’s perfect for expensive computations that are called repeatedly with the same arguments.
The Problem It Solves
Expensive computations called repeatedly waste resources:
# Without caching: recomputes every time
def fibonacci(n):
if n < 2:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
# This is slow for large n
import time
start = time.time()
result = fibonacci(35)
print(f"Took {time.time() - start:.2f} seconds") # Takes several seconds!
lru_cache solves this by remembering previous results.
Basic Usage
from functools import lru_cache
@lru_cache(maxsize=128)
def fibonacci(n):
"""Calculate fibonacci number with caching"""
if n < 2:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
# First call computes the result
import time
start = time.time()
result = fibonacci(35)
print(f"First call: {time.time() - start:.4f} seconds") # Fast!
# Second call uses cached result
start = time.time()
result = fibonacci(35)
print(f"Second call: {time.time() - start:.6f} seconds") # Nearly instant!
Syntax and Parameters
@lru_cache(maxsize=128, typed=False)
def function(*args, **kwargs):
pass
maxsize: Maximum number of cached results (None = unlimited)typed: If True, arguments of different types are cached separately
Practical Example 1: API Response Caching
from functools import lru_cache
import time
@lru_cache(maxsize=32)
def fetch_user_data(user_id):
"""Fetch user data from API (simulated)"""
print(f"Fetching user {user_id} from API...")
time.sleep(1) # Simulate API call
return {'id': user_id, 'name': f'User {user_id}'}
# First call fetches from API
start = time.time()
user1 = fetch_user_data(1)
print(f"First call took {time.time() - start:.2f}s") # ~1 second
# Second call uses cache
start = time.time()
user1_again = fetch_user_data(1)
print(f"Second call took {time.time() - start:.4f}s") # ~0 seconds
# Different user fetches from API
start = time.time()
user2 = fetch_user_data(2)
print(f"Different user took {time.time() - start:.2f}s") # ~1 second
Practical Example 2: Expensive Computation
from functools import lru_cache
import math
@lru_cache(maxsize=128)
def is_prime(n):
"""Check if a number is prime"""
if n < 2:
return False
if n == 2:
return True
if n % 2 == 0:
return False
for i in range(3, int(math.sqrt(n)) + 1, 2):
if n % i == 0:
return False
return True
# Check many numbers
numbers = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29] * 100
import time
start = time.time()
results = [is_prime(n) for n in numbers]
elapsed = time.time() - start
print(f"Checked {len(numbers)} numbers in {elapsed:.4f}s")
print(f"Cache info: {is_prime.cache_info()}")
# Output: CacheInfo(hits=990, misses=10, maxsize=128, currsize=10)
Practical Example 3: Typed Caching
from functools import lru_cache
@lru_cache(maxsize=128, typed=True)
def add(a, b):
"""Add two numbers"""
print(f"Computing {a} + {b}")
return a + b
# Without typed=True, these would share cache
print(add(1, 2)) # Computing 1 + 2, Output: 3
print(add(1, 2)) # Output: 3 (cached)
print(add(1.0, 2.0)) # Computing 1.0 + 2.0, Output: 3.0 (different cache entry)
print(add(1.0, 2.0)) # Output: 3.0 (cached)
print(add.cache_info())
# Output: CacheInfo(hits=2, misses=2, maxsize=128, currsize=2)
Cache Management
from functools import lru_cache
@lru_cache(maxsize=128)
def expensive_function(x):
return x ** 2
# Get cache statistics
print(expensive_function.cache_info())
# Output: CacheInfo(hits=0, misses=0, maxsize=128, currsize=0)
# Call the function
expensive_function(5)
expensive_function(5) # Cache hit
expensive_function(10)
print(expensive_function.cache_info())
# Output: CacheInfo(hits=1, misses=2, maxsize=128, currsize=2)
# Clear the cache
expensive_function.cache_clear()
print(expensive_function.cache_info())
# Output: CacheInfo(hits=0, misses=0, maxsize=128, currsize=0)
When to Use lru_cache
- Expensive computations called repeatedly
- API calls with the same parameters
- Database queries
- Mathematical calculations
- Any pure function with expensive operations
Important Considerations
# ❌ Don't use with mutable arguments
@lru_cache(maxsize=128)
def process_list(items):
# items must be hashable, so use tuple instead
return sum(items)
# ✓ Use with immutable arguments
@lru_cache(maxsize=128)
def process_tuple(items):
return sum(items)
# ❌ Don't use with functions that have side effects
@lru_cache(maxsize=128)
def get_current_time():
import time
return time.time() # Returns different values, caching is wrong!
# ✓ Use with pure functions only
@lru_cache(maxsize=128)
def calculate_tax(amount, rate):
return amount * rate
Combining functools Functions
These functions work well together:
from functools import partial, wraps, lru_cache
def cached_api_call(endpoint, base_url='https://api.example.com', cache_size=32):
"""Create a cached API call function"""
@lru_cache(maxsize=cache_size)
@wraps(lambda: None) # Placeholder for wraps
def fetch(resource_id):
"""Fetch resource from API"""
url = f"{base_url}{endpoint}/{resource_id}"
# Simulate API call
return {'id': resource_id, 'data': 'example'}
return fetch
# Create specialized cached functions
get_users = cached_api_call('/users')
get_products = cached_api_call('/products')
# Use them
user = get_users(1)
user_again = get_users(1) # Uses cache
Best Practices
1. Use partial for Configuration
from functools import partial
# ✓ Good: Clear intent
format_usd = partial(format_currency, currency='USD')
# ❌ Avoid: Unclear
f = partial(format_currency, 'USD')
2. Always Use wraps in Decorators
from functools import wraps
# ✓ Good: Preserves metadata
def my_decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
# ❌ Bad: Loses metadata
def my_decorator(func):
def wrapper(*args, **kwargs):
return func(*args, **kwargs)
return wrapper
3. Use lru_cache Only for Pure Functions
from functools import lru_cache
# ✓ Good: Pure function
@lru_cache(maxsize=128)
def calculate_discount(price, discount_rate):
return price * (1 - discount_rate)
# ❌ Bad: Not pure (depends on external state)
@lru_cache(maxsize=128)
def get_user_discount(user_id):
return current_user.discount_rate # Depends on external state
4. Monitor Cache Performance
from functools import lru_cache
@lru_cache(maxsize=128)
def expensive_function(x):
return x ** 2
# Use cache_info to monitor performance
expensive_function(5)
expensive_function(5)
expensive_function(10)
info = expensive_function.cache_info()
hit_rate = info.hits / (info.hits + info.misses) if (info.hits + info.misses) > 0 else 0
print(f"Cache hit rate: {hit_rate:.1%}")
Conclusion
The functools module provides powerful utilities that solve common programming challenges:
partialcreates specialized functions from general ones, reducing code repetition and improving readabilitywrapspreserves function metadata in decorators, maintaining introspection and documentationlru_cachecaches function results for performance, dramatically speeding up repeated computations
These three functions are essential tools for writing clean, efficient Python code. Understanding when and how to use them will make you a more effective developer.
Key takeaways:
- Use
partialto create specialized versions of functions - Always use
wrapswhen creating decorators - Use
lru_cachefor expensive pure functions - Monitor cache performance with
cache_info() - Remember that
lru_cacheonly works with pure functions
Start incorporating these utilities into your projects today. You’ll find that they solve real problems and make your code more elegant and efficient.
Comments