Two of the most essential skills for Python developers are working with files and optimizing function behavior. The functools module provides powerful tools for function optimization and composition, while file I/O operations are fundamental to almost every real-world application. Understanding both deeply will make you a more capable and efficient developer.
This comprehensive guide covers both topics, providing practical examples and best practices you can apply immediately in your projects.
Part 1: The functools Module
What is functools?
The functools module is part of Python’s standard library and provides higher-order functions and operations on callable objects. It’s designed to help you write cleaner, more efficient code by providing utilities for function composition, caching, and partial application.
Key Functions in functools
The module contains several useful functions, but we’ll focus on the most practical ones: lru_cache, wraps, partial, and reduce.
lru_cache: Caching Function Results
Understanding lru_cache
lru_cache (Least Recently Used cache) is a decorator that caches function results based on arguments. When a function is called with the same arguments again, the cached result is returned instead of recomputing.
Basic Example
from functools import lru_cache
import time
# Without caching
def fibonacci_slow(n):
"""Calculate fibonacci number without caching"""
if n < 2:
return n
return fibonacci_slow(n - 1) + fibonacci_slow(n - 2)
# With caching
@lru_cache(maxsize=128)
def fibonacci_fast(n):
"""Calculate fibonacci number with caching"""
if n < 2:
return n
return fibonacci_fast(n - 1) + fibonacci_fast(n - 2)
# Compare performance
start = time.time()
result_slow = fibonacci_slow(30)
slow_time = time.time() - start
start = time.time()
result_fast = fibonacci_fast(30)
fast_time = time.time() - start
print(f"Without cache: {slow_time:.4f}s")
print(f"With cache: {fast_time:.6f}s")
print(f"Speedup: {slow_time / fast_time:.0f}x faster")
Cache Parameters and Management
from functools import lru_cache
@lru_cache(maxsize=128, typed=False)
def expensive_computation(x, y):
"""Perform expensive computation"""
return x ** y
# Call the function
result1 = expensive_computation(2, 10)
result2 = expensive_computation(2, 10) # Uses cache
result3 = expensive_computation(3, 10) # New computation
# Check cache statistics
info = expensive_computation.cache_info()
print(f"Cache info: {info}")
# Output: CacheInfo(hits=1, misses=2, maxsize=128, currsize=2)
# Clear the cache if needed
expensive_computation.cache_clear()
print(expensive_computation.cache_info())
# Output: CacheInfo(hits=0, misses=0, maxsize=128, currsize=0)
Practical Use Case: API Response Caching
from functools import lru_cache
import time
@lru_cache(maxsize=32)
def fetch_user_data(user_id):
"""Fetch user data from API (simulated)"""
print(f"Fetching user {user_id} from API...")
time.sleep(1) # Simulate network delay
return {
'id': user_id,
'name': f'User {user_id}',
'email': f'user{user_id}@example.com'
}
# First call - fetches from API
user1 = fetch_user_data(1)
print(f"First call took ~1 second")
# Second call - uses cache
user1_again = fetch_user_data(1)
print(f"Second call was instant (cached)")
# Different user - fetches from API
user2 = fetch_user_data(2)
print(f"Different user took ~1 second")
wraps: Preserving Function Metadata
Understanding wraps
When you create a decorator, the wrapper function replaces the original function’s metadata. wraps is a decorator that copies important attributes from the wrapped function to the wrapper.
The Problem It Solves
# Without wraps - metadata is lost
def simple_decorator(func):
def wrapper(*args, **kwargs):
"""Wrapper function"""
return func(*args, **kwargs)
return wrapper
@simple_decorator
def calculate_sum(numbers):
"""Calculate the sum of numbers"""
return sum(numbers)
print(calculate_sum.__name__) # Output: wrapper (should be calculate_sum!)
print(calculate_sum.__doc__) # Output: Wrapper function (should be the original doc!)
Using wraps Correctly
from functools import wraps
def proper_decorator(func):
@wraps(func) # Preserves metadata
def wrapper(*args, **kwargs):
"""Wrapper function"""
return func(*args, **kwargs)
return wrapper
@proper_decorator
def calculate_sum(numbers):
"""Calculate the sum of numbers"""
return sum(numbers)
print(calculate_sum.__name__) # Output: calculate_sum
print(calculate_sum.__doc__) # Output: Calculate the sum of numbers
Practical Example: Logging Decorator
from functools import wraps
import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
def log_function_call(func):
"""Decorator that logs function calls"""
@wraps(func)
def wrapper(*args, **kwargs):
logger.info(f"Calling {func.__name__} with args={args}, kwargs={kwargs}")
try:
result = func(*args, **kwargs)
logger.info(f"{func.__name__} returned {result}")
return result
except Exception as e:
logger.error(f"{func.__name__} raised {type(e).__name__}: {e}")
raise
return wrapper
@log_function_call
def divide(a, b):
"""Divide two numbers"""
return a / b
# Metadata is preserved
print(f"Function name: {divide.__name__}")
print(f"Function doc: {divide.__doc__}")
# Function works with logging
result = divide(10, 2)
partial: Creating Specialized Functions
Understanding partial
partial creates a new function by fixing some arguments of an existing function. It’s useful for creating specialized versions of general functions.
Basic Example
from functools import partial
def multiply(a, b):
"""Multiply two numbers"""
return a * b
# Create specialized functions
double = partial(multiply, b=2)
triple = partial(multiply, b=3)
print(double(5)) # Output: 10
print(triple(5)) # Output: 15
Practical Example: Data Formatting
from functools import partial
def format_currency(value, currency='USD', decimal_places=2):
"""Format a number as currency"""
formatted = f"{value:,.{decimal_places}f}"
return f"{currency} {formatted}"
# Create specialized formatters
format_usd = partial(format_currency, currency='USD')
format_eur = partial(format_currency, currency='EUR')
format_btc = partial(format_currency, currency='BTC', decimal_places=8)
# Use the specialized formatters
prices = [1000.5, 2500.75, 15000.123]
print("USD prices:")
for price in prices:
print(f" {format_usd(price)}")
print("\nEUR prices:")
for price in prices:
print(f" {format_eur(price)}")
print("\nBTC prices:")
for price in prices:
print(f" {format_btc(price)}")
reduce: Aggregating Values
Understanding reduce
reduce applies a function cumulatively to items in an iterable, from left to right, to reduce it to a single value.
Basic Example
from functools import reduce
# Sum all numbers
numbers = [1, 2, 3, 4, 5]
total = reduce(lambda a, b: a + b, numbers)
print(f"Sum: {total}") # Output: 15
# Find maximum
maximum = reduce(lambda a, b: a if a > b else b, numbers)
print(f"Maximum: {maximum}") # Output: 5
# Multiply all numbers
product = reduce(lambda a, b: a * b, numbers)
print(f"Product: {product}") # Output: 120
Practical Example: Building a Dictionary
from functools import reduce
# Build a dictionary from a list of tuples
items = [('name', 'Alice'), ('age', 30), ('city', 'NYC'), ('job', 'Engineer')]
result = reduce(
lambda d, item: {**d, item[0]: item[1]},
items,
{} # Initial value
)
print(result)
# Output: {'name': 'Alice', 'age': 30, 'city': 'NYC', 'job': 'Engineer'}
Part 2: File I/O Operations
Understanding File Modes
Python supports several file modes for different operations:
| Mode | Description |
|---|---|
'r' |
Read (default) - file must exist |
'w' |
Write - creates new file or truncates existing |
'a' |
Append - adds to end of file |
'r+' |
Read and write - file must exist |
'w+' |
Write and read - creates or truncates file |
'a+' |
Append and read - creates if doesn’t exist |
'b' |
Binary mode (combine with others: 'rb', 'wb') |
Reading Files
Method 1: read() - Read Entire File
# Read entire file at once
with open('example.txt', 'r') as file:
content = file.read()
print(content)
# Be careful with large files - loads entire content into memory
Method 2: readline() - Read One Line
# Read one line at a time
with open('example.txt', 'r') as file:
line1 = file.readline() # First line
line2 = file.readline() # Second line
print(line1)
print(line2)
Method 3: readlines() - Read All Lines as List
# Read all lines into a list
with open('example.txt', 'r') as file:
lines = file.readlines()
for i, line in enumerate(lines, 1):
print(f"Line {i}: {line.rstrip()}")
Method 4: Iteration - Most Pythonic
# Iterate over lines (most memory-efficient for large files)
with open('example.txt', 'r') as file:
for line in file:
print(line.rstrip()) # rstrip() removes trailing newline
Practical Example: Processing Log Files
def count_errors_in_log(filename):
"""Count error occurrences in a log file"""
error_count = 0
error_lines = []
with open(filename, 'r') as file:
for line_num, line in enumerate(file, 1):
if 'ERROR' in line:
error_count += 1
error_lines.append((line_num, line.rstrip()))
return error_count, error_lines
# Usage
# count, errors = count_errors_in_log('app.log')
# print(f"Found {count} errors")
# for line_num, line in errors:
# print(f" Line {line_num}: {line}")
Writing Files
Method 1: write() - Write String
# Write to file (overwrites existing content)
with open('output.txt', 'w') as file:
file.write("Hello, World!\n")
file.write("This is a test file.\n")
Method 2: writelines() - Write Multiple Lines
# Write multiple lines
lines = [
"Line 1\n",
"Line 2\n",
"Line 3\n"
]
with open('output.txt', 'w') as file:
file.writelines(lines)
Method 3: Append Mode - Add to Existing File
# Append to existing file
with open('output.txt', 'a') as file:
file.write("This line is appended.\n")
Practical Example: Generating a Report
def generate_report(data, filename):
"""Generate a report file from data"""
with open(filename, 'w') as file:
# Write header
file.write("=" * 50 + "\n")
file.write("REPORT\n")
file.write("=" * 50 + "\n\n")
# Write data
for item in data:
file.write(f"Name: {item['name']}\n")
file.write(f"Value: {item['value']}\n")
file.write(f"Status: {item['status']}\n")
file.write("-" * 50 + "\n")
# Write footer
file.write(f"\nTotal items: {len(data)}\n")
# Usage
data = [
{'name': 'Item 1', 'value': 100, 'status': 'Active'},
{'name': 'Item 2', 'value': 200, 'status': 'Inactive'},
{'name': 'Item 3', 'value': 150, 'status': 'Active'},
]
# generate_report(data, 'report.txt')
The with Statement: Context Managers
Why Use with?
The with statement ensures files are properly closed, even if an error occurs. It’s the recommended way to handle files.
# ❌ Without with - file might not close if error occurs
file = open('example.txt', 'r')
content = file.read()
file.close()
# ✓ With with - file always closes
with open('example.txt', 'r') as file:
content = file.read()
# File is automatically closed here
Multiple Files
# Work with multiple files
with open('input.txt', 'r') as infile, open('output.txt', 'w') as outfile:
for line in infile:
# Process and write
processed = line.upper()
outfile.write(processed)
Error Handling in File Operations
Common Errors and How to Handle Them
import os
def safe_read_file(filename):
"""Safely read a file with error handling"""
try:
with open(filename, 'r') as file:
return file.read()
except FileNotFoundError:
print(f"Error: File '{filename}' not found")
return None
except PermissionError:
print(f"Error: Permission denied reading '{filename}'")
return None
except IOError as e:
print(f"Error reading file: {e}")
return None
# Usage
content = safe_read_file('example.txt')
if content:
print(content)
Checking File Existence
import os
filename = 'example.txt'
# Check if file exists
if os.path.exists(filename):
print(f"{filename} exists")
else:
print(f"{filename} does not exist")
# Check if it's a file (not a directory)
if os.path.isfile(filename):
print(f"{filename} is a file")
# Get file size
if os.path.exists(filename):
size = os.path.getsize(filename)
print(f"File size: {size} bytes")
Practical Example: CSV-like Data Processing
def read_csv_like(filename, delimiter=','):
"""Read a simple CSV-like file"""
data = []
with open(filename, 'r') as file:
# Read header
header = file.readline().rstrip().split(delimiter)
# Read data rows
for line in file:
values = line.rstrip().split(delimiter)
row = dict(zip(header, values))
data.append(row)
return data
def write_csv_like(filename, data, header, delimiter=','):
"""Write data to a CSV-like file"""
with open(filename, 'w') as file:
# Write header
file.write(delimiter.join(header) + '\n')
# Write data rows
for row in data:
values = [str(row.get(col, '')) for col in header]
file.write(delimiter.join(values) + '\n')
# Usage
data = [
{'name': 'Alice', 'age': '30', 'city': 'NYC'},
{'name': 'Bob', 'age': '25', 'city': 'LA'},
{'name': 'Charlie', 'age': '35', 'city': 'Chicago'},
]
header = ['name', 'age', 'city']
# write_csv_like('data.csv', data, header)
# read_data = read_csv_like('data.csv')
# print(read_data)
Best Practices for File I/O
1. Always Use Context Managers
# ✓ Good
with open('file.txt', 'r') as file:
content = file.read()
# ❌ Avoid
file = open('file.txt', 'r')
content = file.read()
file.close()
2. Handle Exceptions
# ✓ Good
try:
with open('file.txt', 'r') as file:
content = file.read()
except FileNotFoundError:
print("File not found")
except IOError as e:
print(f"Error reading file: {e}")
3. Use Appropriate Read Methods
# For small files
with open('small.txt', 'r') as file:
content = file.read()
# For large files - iterate to save memory
with open('large.txt', 'r') as file:
for line in file:
process(line)
4. Close Files Explicitly When Needed
# If not using with statement
file = open('file.txt', 'r')
try:
content = file.read()
finally:
file.close() # Ensures file closes even if error occurs
5. Use Pathlib for Modern File Operations
from pathlib import Path
# Modern approach using pathlib
file_path = Path('example.txt')
# Check if file exists
if file_path.exists():
# Read file
content = file_path.read_text()
# Write file
file_path.write_text("New content")
# Get file info
print(f"File size: {file_path.stat().st_size} bytes")
Combining functools and File I/O
Here’s a practical example combining both concepts:
from functools import lru_cache, wraps
import os
@lru_cache(maxsize=32)
def read_config_file(filename):
"""Read and cache configuration file"""
with open(filename, 'r') as file:
config = {}
for line in file:
line = line.strip()
if line and not line.startswith('#'):
key, value = line.split('=')
config[key.strip()] = value.strip()
return config
def log_file_operation(func):
"""Decorator to log file operations"""
@wraps(func)
def wrapper(*args, **kwargs):
print(f"Performing file operation: {func.__name__}")
try:
result = func(*args, **kwargs)
print(f"Operation successful")
return result
except Exception as e:
print(f"Operation failed: {e}")
raise
return wrapper
@log_file_operation
def process_file(input_file, output_file):
"""Process input file and write to output"""
with open(input_file, 'r') as infile, open(output_file, 'w') as outfile:
for line in infile:
processed = line.upper()
outfile.write(processed)
# Usage
# config = read_config_file('config.txt')
# process_file('input.txt', 'output.txt')
Conclusion
The functools module and file I/O operations are essential tools for Python developers:
functools provides:
lru_cachefor performance optimization through cachingwrapsfor preserving function metadata in decoratorspartialfor creating specialized functionsreducefor aggregating values
File I/O best practices:
- Always use
withstatements for file operations - Choose the right read method for your use case
- Handle exceptions appropriately
- Use context managers to ensure files close properly
- Consider using
pathlibfor modern file operations
Key takeaways:
- Use
lru_cacheto optimize expensive computations - Always use
@wrapswhen creating decorators - Use
partialto create specialized function versions - Use
withstatements for all file operations - Handle file-related exceptions gracefully
- Choose read methods based on file size and use case
By mastering these tools, you’ll write more efficient, maintainable, and Pythonic code. Start incorporating these practices into your projects today, and you’ll see immediate improvements in code quality and performance.
Comments