Functional Programming Libraries in Python: toolz and fn.py
Python is a multi-paradigm language that supports functional programming, but it doesn’t always make it easy. Functional programming libraries fill this gap, providing utilities that make functional patterns more natural and expressive in Python.
Two standout libraries are toolz and fn.py. Both bring functional programming concepts to Python, but they take different approaches. This guide explores both libraries, showing you how to leverage them to write cleaner, more composable code.
Why Functional Programming Libraries Matter
Python’s built-in functional tools (map, filter, reduce) are powerful but limited. They don’t compose well, they’re verbose, and they don’t integrate smoothly with Python’s imperative style. Functional programming libraries solve these problems by providing:
- Composable utilities: Functions that work together seamlessly
- Concise syntax: Less boilerplate, more expressiveness
- Lazy evaluation: Process large datasets efficiently
- Immutable data structures: Safer, more predictable code
- Functional patterns: Currying, partial application, and more
toolz: Functional Utilities for Python
What is toolz?
toolz is a Python library that provides a set of utility functions for working with iterators, functions, and dictionaries. It’s designed to make functional programming patterns natural and efficient in Python.
Installation
pip install toolz
Key Features
toolz provides several categories of utilities:
- Iterator functions:
partition,pluck,groupby,unique,frequencies - Function composition:
compose,pipe,juxt - Dictionary operations:
merge,valmap,keymap,itemmap - Functional utilities:
curry,memoize,flip
Working with Iterators
Partitioning Data
from toolz import partition
# Split data into chunks
data = [1, 2, 3, 4, 5, 6, 7, 8, 9]
pairs = list(partition(2, data))
print(pairs) # [(1, 2), (3, 4), (5, 6), (7, 8), (9,)]
# Useful for batch processing
def process_batch(batch):
return sum(batch)
batches = partition(3, data)
results = [process_batch(batch) for batch in batches]
print(results) # [6, 15, 24]
Extracting Values with pluck
from toolz import pluck
# Extract specific fields from dictionaries
users = [
{'name': 'Alice', 'age': 30, 'city': 'NYC'},
{'name': 'Bob', 'age': 25, 'city': 'LA'},
{'name': 'Charlie', 'age': 35, 'city': 'Chicago'},
]
# Get all names
names = list(pluck('name', users))
print(names) # ['Alice', 'Bob', 'Charlie']
# Get multiple fields
name_age = list(pluck(['name', 'age'], users))
print(name_age) # [('Alice', 30), ('Bob', 25), ('Charlie', 35)]
Grouping and Frequencies
from toolz import groupby, frequencies
# Group by a key function
users = [
{'name': 'Alice', 'department': 'Engineering'},
{'name': 'Bob', 'department': 'Sales'},
{'name': 'Charlie', 'department': 'Engineering'},
{'name': 'Diana', 'department': 'Sales'},
]
by_dept = groupby('department', users)
print(dict(by_dept))
# {
# 'Engineering': [{'name': 'Alice', ...}, {'name': 'Charlie', ...}],
# 'Sales': [{'name': 'Bob', ...}, {'name': 'Diana', ...}]
# }
# Count frequencies
words = ['apple', 'banana', 'apple', 'cherry', 'banana', 'apple']
freq = frequencies(words)
print(freq) # {'apple': 3, 'banana': 2, 'cherry': 1}
Finding Unique Elements
from toolz import unique
# Remove duplicates while preserving order
data = [1, 2, 2, 3, 1, 4, 3, 5]
unique_data = list(unique(data))
print(unique_data) # [1, 2, 3, 4, 5]
# Works with any iterable
words = ['apple', 'banana', 'apple', 'cherry']
unique_words = list(unique(words))
print(unique_words) # ['apple', 'banana', 'cherry']
Function Composition
Composing Functions
from toolz import compose, pipe
# Right-to-left composition (mathematical style)
add_one = lambda x: x + 1
double = lambda x: x * 2
square = lambda x: x ** 2
f = compose(square, double, add_one)
result = f(5) # ((5 + 1) * 2) ^ 2 = 144
print(result)
# Left-to-right piping (more readable)
result = pipe(5, add_one, double, square)
print(result) # 144
Using juxt for Multiple Functions
from toolz import juxt
# Apply multiple functions to the same input
data = [1, 2, 3, 4, 5]
# Get min, max, and sum
stats = juxt(min, max, sum)
result = stats(data)
print(result) # (1, 5, 15)
# Useful for computing multiple statistics
def get_stats(numbers):
min_val, max_val, total = juxt(min, max, sum)(numbers)
return {
'min': min_val,
'max': max_val,
'sum': total,
'avg': total / len(numbers),
}
print(get_stats([1, 2, 3, 4, 5]))
# {'min': 1, 'max': 5, 'sum': 15, 'avg': 3.0}
Dictionary Operations
from toolz import merge, valmap, keymap, itemmap
# Merge multiple dictionaries
dict1 = {'a': 1, 'b': 2}
dict2 = {'c': 3, 'd': 4}
merged = merge(dict1, dict2)
print(merged) # {'a': 1, 'b': 2, 'c': 3, 'd': 4}
# Transform values
prices = {'apple': 1.5, 'banana': 0.75, 'cherry': 2.0}
discounted = valmap(lambda x: x * 0.9, prices)
print(discounted) # {'apple': 1.35, 'banana': 0.675, 'cherry': 1.8}
# Transform keys
data = {'first_name': 'Alice', 'last_name': 'Smith'}
camel_case = keymap(lambda k: k.replace('_', ''), data)
print(camel_case) # {'firstname': 'Alice', 'lastname': 'Smith'}
# Transform items
items = {'a': 1, 'b': 2, 'c': 3}
transformed = itemmap(lambda k, v: (k.upper(), v * 2), items)
print(transformed) # {'A': 2, 'B': 4, 'C': 6}
Functional Utilities
Currying Functions
from toolz import curry
# Create a curried function
@curry
def add(x, y, z):
return x + y + z
# Can be called with all arguments at once
result1 = add(1, 2, 3)
print(result1) # 6
# Or partially applied
add_one = add(1)
add_one_two = add_one(2)
result2 = add_one_two(3)
print(result2) # 6
# Useful for creating specialized functions
@curry
def multiply(x, y):
return x * y
double = multiply(2)
triple = multiply(3)
print(double(5)) # 10
print(triple(5)) # 15
Memoization
from toolz import memoize
# Cache function results
@memoize
def fibonacci(n):
if n < 2:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
# First call computes
result1 = fibonacci(10)
print(result1) # 55
# Subsequent calls use cache
result2 = fibonacci(10)
print(result2) # 55 (from cache)
# Check cache info
print(fibonacci.cache)
Real-World Example: Data Processing Pipeline
from toolz import compose, pluck, groupby, valmap, frequencies
# Sample data
transactions = [
{'user': 'Alice', 'amount': 100, 'category': 'food'},
{'user': 'Bob', 'amount': 50, 'category': 'transport'},
{'user': 'Alice', 'amount': 75, 'category': 'entertainment'},
{'user': 'Charlie', 'amount': 200, 'category': 'food'},
{'user': 'Bob', 'amount': 30, 'category': 'food'},
]
# Build a processing pipeline
def analyze_transactions(transactions):
# Group by user
by_user = groupby('user', transactions)
# Calculate total spending per user
user_totals = valmap(
lambda txns: sum(t['amount'] for t in txns),
by_user
)
# Get categories
categories = list(pluck('category', transactions))
category_freq = frequencies(categories)
return {
'user_totals': dict(user_totals),
'category_frequency': dict(category_freq),
}
result = analyze_transactions(transactions)
print(result)
# {
# 'user_totals': {'Alice': 175, 'Bob': 80, 'Charlie': 200},
# 'category_frequency': {'food': 3, 'transport': 1, 'entertainment': 1}
# }
fn.py: Functional Programming for Python
What is fn.py?
fn.py is a library that brings functional programming features to Python, including improved lambda syntax, streams, and functional composition. It’s designed to make functional programming more natural and expressive.
Installation
pip install fn
Key Features
fn.py provides:
- Streams: Lazy, composable sequences
- Functional operators:
_for lambda syntax,*for unpacking - Composition:
composeandpipeutilities - Immutable data structures: Persistent collections
- Functional utilities:
curry,partial, and more
Working with Streams
Creating and Filtering Streams
from fn import Stream
# Create a stream from a list
numbers = Stream(range(1, 11))
# Filter and map
result = (numbers
.filter(lambda x: x % 2 == 0)
.map(lambda x: x ** 2)
.to_list())
print(result) # [4, 16, 36, 64, 100]
# Streams are lazy - nothing is computed until you consume them
stream = Stream(range(1, 1000000))
result = (stream
.filter(lambda x: x % 2 == 0)
.map(lambda x: x ** 2)
.take(5)
.to_list())
print(result) # [4, 16, 36, 64, 100] - only computed what's needed
Using the _ Operator for Cleaner Lambdas
from fn import _
# Traditional lambda
result1 = list(map(lambda x: x * 2, [1, 2, 3, 4, 5]))
# Using _ operator (much cleaner!)
result2 = list(map(_ * 2, [1, 2, 3, 4, 5]))
print(result1) # [2, 4, 6, 8, 10]
print(result2) # [2, 4, 6, 8, 10]
# Works with complex expressions
data = [
{'name': 'Alice', 'age': 30},
{'name': 'Bob', 'age': 25},
{'name': 'Charlie', 'age': 35},
]
# Extract names using _
names = list(map(_['name'], data))
print(names) # ['Alice', 'Bob', 'Charlie']
# Filter using _
adults = list(filter(_ ['age'] >= 30, data))
print(adults) # [{'name': 'Alice', 'age': 30}, {'name': 'Charlie', 'age': 35}]
Stream Operations
from fn import Stream
# Chaining operations
numbers = Stream(range(1, 11))
result = (numbers
.filter(lambda x: x > 3)
.map(lambda x: x * 2)
.reduce(lambda x, y: x + y))
print(result) # 2 + 4 + 6 + 8 + 10 + 12 + 14 + 16 + 18 + 20 = 110
# Group by
data = [
{'category': 'A', 'value': 10},
{'category': 'B', 'value': 20},
{'category': 'A', 'value': 15},
{'category': 'B', 'value': 25},
]
grouped = Stream(data).group_by(lambda x: x['category'])
print(dict(grouped))
# {
# 'A': [{'category': 'A', 'value': 10}, {'category': 'A', 'value': 15}],
# 'B': [{'category': 'B', 'value': 20}, {'category': 'B', 'value': 25}]
# }
Functional Composition
from fn import compose, pipe
# Compose functions
add_one = lambda x: x + 1
double = lambda x: x * 2
square = lambda x: x ** 2
# Right-to-left composition
f = compose(square, double, add_one)
result = f(5) # ((5 + 1) * 2) ^ 2 = 144
print(result)
# Left-to-right piping
result = pipe(5, add_one, double, square)
print(result) # 144
Using the _ Operator with Operators
from fn import _
# Arithmetic operations
add = _ + 5
multiply = _ * 3
divide = _ / 2
print(add(10)) # 15
print(multiply(10)) # 30
print(divide(10)) # 5.0
# Comparison operations
is_positive = _ > 0
is_even = _ % 2 == 0
numbers = [1, -2, 3, -4, 5, 6]
positive = list(filter(is_positive, numbers))
even = list(filter(is_even, numbers))
print(positive) # [1, 3, 5, 6]
print(even) # [-2, -4, 6]
# Attribute access
get_name = _['name']
get_age = _['age']
users = [
{'name': 'Alice', 'age': 30},
{'name': 'Bob', 'age': 25},
]
names = list(map(get_name, users))
ages = list(map(get_age, users))
print(names) # ['Alice', 'Bob']
print(ages) # [30, 25]
Real-World Example: Data Analysis Pipeline
from fn import Stream, _
# Sample data
sales = [
{'product': 'Laptop', 'price': 1000, 'quantity': 2, 'region': 'US'},
{'product': 'Mouse', 'price': 25, 'quantity': 10, 'region': 'US'},
{'product': 'Keyboard', 'price': 75, 'quantity': 5, 'region': 'EU'},
{'product': 'Monitor', 'price': 300, 'quantity': 3, 'region': 'US'},
{'product': 'Laptop', 'price': 1000, 'quantity': 1, 'region': 'EU'},
]
# Calculate revenue per item
def add_revenue(item):
return {**item, 'revenue': item['price'] * item['quantity']}
# Build analysis pipeline
analysis = (Stream(sales)
.map(add_revenue)
.filter(_['revenue'] > 100)
.group_by(_['region'])
.map(lambda kv: {
'region': kv[0],
'total_revenue': sum(item['revenue'] for item in kv[1]),
'items': len(kv[1])
})
.to_list())
print(analysis)
# [
# {'region': 'US', 'total_revenue': 3150, 'items': 3},
# {'region': 'EU', 'total_revenue': 1375, 'items': 2}
# ]
Comparing toolz and fn.py
| Feature | toolz | fn.py |
|---|---|---|
| Focus | Iterator and function utilities | Streams and functional operators |
| Lambda Syntax | Standard Python lambdas | _ operator for cleaner syntax |
| Lazy Evaluation | Limited (some functions) | Full support via Streams |
| Dictionary Operations | Excellent (valmap, keymap, etc.) |
Limited |
| Composition | compose, pipe |
compose, pipe |
| Learning Curve | Gentle | Moderate (due to _ operator) |
| Use Case | Data transformation, utilities | Stream processing, functional chains |
| Performance | Optimized for iterators | Optimized for lazy evaluation |
When to Use Each Library
Use toolz When:
- You need to transform dictionaries and iterators
- You want a lightweight, focused library
- You’re doing data extraction and grouping
- You need function composition with clear semantics
- You prefer explicit, readable code
Use fn.py When:
- You’re processing large datasets lazily
- You want cleaner lambda syntax with
_ - You’re building complex functional pipelines
- You need stream-based processing
- You want to minimize intermediate collections
Use Both Together
You can combine both libraries for maximum power:
from toolz import groupby, valmap, pluck
from fn import Stream, _
# Use toolz for initial grouping and transformation
data = [
{'user': 'Alice', 'score': 85},
{'user': 'Bob', 'score': 92},
{'user': 'Alice', 'score': 88},
{'user': 'Charlie', 'score': 78},
]
by_user = groupby('user', data)
user_scores = valmap(lambda scores: list(pluck('score', scores)), by_user)
# Use fn.py for stream processing
result = (Stream(user_scores.items())
.map(lambda kv: {'user': kv[0], 'avg_score': sum(kv[1]) / len(kv[1])})
.filter(_['avg_score'] >= 85)
.to_list())
print(result)
# [{'user': 'Alice', 'avg_score': 86.5}]
Best Practices
1. Use Composition for Clarity
# Good: Clear pipeline
from toolz import compose
process = compose(
list,
sorted,
set,
)
result = process([3, 1, 2, 1, 3])
print(result) # [1, 2, 3]
# Less clear: Nested calls
result = sorted(set([3, 1, 2, 1, 3]))
2. Leverage Lazy Evaluation
from fn import Stream
# Good: Only compute what's needed
result = (Stream(range(1000000))
.filter(lambda x: x % 2 == 0)
.map(lambda x: x ** 2)
.take(5)
.to_list())
# Less efficient: Compute everything
result = [x ** 2 for x in range(1000000) if x % 2 == 0][:5]
3. Use _ for Readable Filters
from fn import _
# Good: Clear intent
adults = list(filter(_ ['age'] >= 18, users))
# Less clear: Lambda
adults = list(filter(lambda u: u['age'] >= 18, users))
4. Combine with Type Hints
from typing import List, Dict
from toolz import pluck
def extract_names(users: List[Dict]) -> List[str]:
return list(pluck('name', users))
Conclusion
toolz and fn.py are powerful libraries that bring functional programming patterns to Python. While they serve different purposes, both help you write cleaner, more composable, and more maintainable code.
Key takeaways:
- toolz excels at iterator and dictionary operations with excellent composition utilities
- fn.py provides lazy streams and cleaner lambda syntax with the
_operator - Composition makes code more readable and testable
- Lazy evaluation enables efficient processing of large datasets
- Combine both libraries for maximum expressiveness
- Start simple with basic operations and gradually adopt more advanced patterns
Whether you’re processing data, building pipelines, or transforming collections, these libraries will help you write more functional, expressive Python code. Start with the library that best fits your use case, and explore the other as your functional programming skills grow.
Comments