Introduction
Testing is fundamental to software quality and developer confidence. A well-tested codebase enables fearless refactoring, catches bugs early, serves as living documentation, and reduces production incidents. Yet testing is often misunderstood—teams either over-test (slow, brittle tests) or under-test (bugs in production).
This guide covers testing strategies that balance speed, reliability, and coverage. We’ll explore the test pyramid, different test types (unit, integration, E2E), test doubles (mocks, stubs, fakes), property-based testing, mutation testing, and best practices for building maintainable test suites.
Key benefits of effective testing:
- Confidence: Deploy without fear
- Regression Prevention: Catch bugs before production
- Documentation: Tests show how code should be used
- Design Feedback: Hard-to-test code is often poorly designed
- Refactoring Safety: Change code without breaking functionality
The Test Pyramid
The test pyramid guides test distribution: many fast unit tests at the base, fewer integration tests in the middle, and few slow E2E tests at the top.
/\
/ \ E2E Tests (Few, Slow, Brittle)
/ \ - Full system tests
/------\ - Browser automation
/ \ - API integration tests
/ \
/------------\ Integration Tests (Some, Medium Speed)
/ \ - Database tests
/ \ - External service tests
/------------------\ Unit Tests (Many, Fast, Stable)
- Pure functions
- Business logic
- Domain models
Why this distribution?
- Unit tests are fast (milliseconds) — run them constantly
- Integration tests are slower (seconds) — run before commit
- E2E tests are slowest (minutes) — run in CI/CD
Anti-pattern: Ice Cream Cone
/\
/ \
/ \ Many E2E tests (slow, brittle)
/ \
/--------\
/ \ Few integration tests
/ \
/ \ Very few unit tests
/----------------\
This inverted pyramid leads to slow test suites, flaky tests, and low developer productivity.
# tests/test_user_service.py
import pytest
from unittest.mock import Mock, patch
from datetime import datetime
class TestUserService:
"""Unit tests for user service."""
@pytest.fixture
def user_repo(self):
return Mock()
@pytest.fixture
def email_service(self):
return Mock()
@pytest.fixture
def service(self, user_repo, email_service):
from services.user_service import UserService
return UserService(user_repo, email_service)
def test_create_user_success(self, service, user_repo, email_service):
"""Test successful user creation."""
user_repo.find_by_email.return_value = None
result = service.create_user(
email="[email protected]",
name="Test User",
password="secure123"
)
assert result.email == "[email protected]"
assert result.name == "Test User"
assert result.id is not None
user_repo.save.assert_called_once()
email_service.send_welcome.assert_called_once()
def test_create_user_duplicate_email(self, service, user_repo):
"""Test user creation with existing email."""
user_repo.find_by_email.return_value = Mock(id="existing")
with pytest.raises(ValueError, match="already exists"):
service.create_user(
email="[email protected]",
name="Test",
password="password123"
)
def test_create_user_invalid_email(self, service):
"""Test user creation with invalid email."""
with pytest.raises(ValueError, match="valid email"):
service.create_user(
email="invalid",
name="Test",
password="password123"
)
# tests/test_api.py
from fastapi.testclient import TestClient
class TestUserAPI:
"""Integration tests for user API."""
@pytest.fixture
def client(self, test_db):
from main import app
from dependencies import get_db
from database import get_engine
TestingSessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=get_engine())
def override_get_db():
db = TestingSessionLocal()
try:
yield db
finally:
db.close()
app.dependency_overrides[get_db] = override_get_db
with TestClient(app) as client:
yield client
app.dependency_overrides.clear()
def test_create_and_get_user(self, client):
"""Test creating and retrieving a user."""
# Create user
response = client.post(
"/api/v1/users",
json={
"email": "[email protected]",
"name": "API Test",
"password": "password123"
}
)
assert response.status_code == 201
user_id = response.json()["id"]
# Get user
response = client.get(f"/api/v1/users/{user_id}")
assert response.status_code == 200
assert response.json()["email"] == "[email protected]"
Mocking and Fixtures
# tests/conftest.py
import pytest
from unittest.mock import Mock, MagicMock
from datetime import datetime, timedelta
@pytest.fixture
def mock_user():
"""Create a mock user object."""
user = Mock()
user.id = "test-id"
user.email = "[email protected]"
user.name = "Test User"
user.is_active = True
user.created_at = datetime.utcnow()
return user
@pytest.fixture
def mock_db_session():
"""Create a mock database session."""
session = MagicMock()
session.query.return_value.filter.return_value.first.return_value = None
return session
@pytest.fixture
def sample_users():
"""Create sample users for testing."""
return [
{"id": "1", "email": "[email protected]", "name": "User 1"},
{"id": "2", "email": "[email protected]", "name": "User 2"},
{"id": "3", "email": "[email protected]", "name": "User 3"},
]
class TestMockingPatterns:
"""Examples of common mocking patterns."""
def test_method_call_verification(self, mock_user):
"""Verify a method was called."""
service = Mock()
service.create_user.return_value = mock_user
result = service.create_user("[email protected]", "Test")
service.create_user.assert_called_once_with("[email protected]", "Test")
assert result.email == "[email protected]"
def test_return_value_side_effect(self, mock_user):
"""Use side effect for complex behavior."""
service = Mock()
service.get_user.side_effect = [
None, # First call returns None
mock_user, # Second call returns user
]
assert service.get_user("1") is None
assert service.get_user("2").email == "[email protected]"
def test_property_mock(self):
"""Mock properties and attributes."""
mock = Mock()
mock.user.name = "Test"
mock.user.email = "[email protected]"
assert mock.user.name == "Test"
Conclusion
Effective testing requires strategy: write many fast unit tests, some integration tests, and few E2E tests. Mock external dependencies. Use fixtures for setup. Test behavior, not implementation. Aim for meaningful coverage, not arbitrary numbers.
Resources
- pytest Documentation
- “Working Effectively with Legacy Code” by Michael Feathers
- Test-Driven Development by Kent Beck
Unit Testing: Testing in Isolation
Unit tests verify individual functions, methods, or classes in isolation. They’re fast, focused, and form the foundation of your test suite.
Characteristics of Good Unit Tests
- Fast: Run in milliseconds
- Isolated: No external dependencies (database, network, filesystem)
- Repeatable: Same result every time
- Self-validating: Pass or fail, no manual inspection
- Timely: Written with or before the code
Unit Test Example (Python)
import pytest
from decimal import Decimal
from datetime import datetime, timedelta
# Code under test
class ShoppingCart:
def __init__(self):
self.items = []
def add_item(self, product_id: str, name: str, price: Decimal, quantity: int):
if quantity <= 0:
raise ValueError("Quantity must be positive")
if price < 0:
raise ValueError("Price cannot be negative")
self.items.append({
'product_id': product_id,
'name': name,
'price': price,
'quantity': quantity
})
def get_total(self) -> Decimal:
return sum(item['price'] * item['quantity'] for item in self.items)
def apply_discount(self, percentage: Decimal) -> Decimal:
if not 0 <= percentage <= 100:
raise ValueError("Discount must be between 0 and 100")
total = self.get_total()
discount = total * (percentage / 100)
return total - discount
def is_empty(self) -> bool:
return len(self.items) == 0
# Unit tests
class TestShoppingCart:
"""Unit tests for ShoppingCart."""
def test_new_cart_is_empty(self):
"""Test that new cart has no items."""
cart = ShoppingCart()
assert cart.is_empty()
assert cart.get_total() == Decimal('0')
def test_add_item_increases_total(self):
"""Test adding item increases cart total."""
cart = ShoppingCart()
cart.add_item('prod-1', 'Widget', Decimal('10.00'), 2)
assert not cart.is_empty()
assert cart.get_total() == Decimal('20.00')
def test_add_multiple_items(self):
"""Test adding multiple items."""
cart = ShoppingCart()
cart.add_item('prod-1', 'Widget', Decimal('10.00'), 2)
cart.add_item('prod-2', 'Gadget', Decimal('15.00'), 1)
assert cart.get_total() == Decimal('35.00')
def test_add_item_with_zero_quantity_raises_error(self):
"""Test that zero quantity raises ValueError."""
cart = ShoppingCart()
with pytest.raises(ValueError, match="Quantity must be positive"):
cart.add_item('prod-1', 'Widget', Decimal('10.00'), 0)
def test_add_item_with_negative_price_raises_error(self):
"""Test that negative price raises ValueError."""
cart = ShoppingCart()
with pytest.raises(ValueError, match="Price cannot be negative"):
cart.add_item('prod-1', 'Widget', Decimal('-10.00'), 1)
def test_apply_discount(self):
"""Test applying discount to cart."""
cart = ShoppingCart()
cart.add_item('prod-1', 'Widget', Decimal('100.00'), 1)
discounted = cart.apply_discount(Decimal('20'))
assert discounted == Decimal('80.00')
def test_apply_discount_over_100_raises_error(self):
"""Test that discount over 100% raises error."""
cart = ShoppingCart()
cart.add_item('prod-1', 'Widget', Decimal('100.00'), 1)
with pytest.raises(ValueError, match="between 0 and 100"):
cart.apply_discount(Decimal('150'))
@pytest.mark.parametrize("price,quantity,expected", [
(Decimal('10.00'), 1, Decimal('10.00')),
(Decimal('10.00'), 5, Decimal('50.00')),
(Decimal('9.99'), 3, Decimal('29.97')),
(Decimal('0.01'), 100, Decimal('1.00')),
])
def test_total_calculation(self, price, quantity, expected):
"""Test total calculation with various inputs."""
cart = ShoppingCart()
cart.add_item('prod-1', 'Widget', price, quantity)
assert cart.get_total() == expected
Unit Test Example (JavaScript/TypeScript)
// Code under test
export class UserValidator {
validateEmail(email: string): boolean {
const emailRegex = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
return emailRegex.test(email);
}
validatePassword(password: string): { valid: boolean; errors: string[] } {
const errors: string[] = [];
if (password.length < 8) {
errors.push('Password must be at least 8 characters');
}
if (!/[A-Z]/.test(password)) {
errors.push('Password must contain uppercase letter');
}
if (!/[a-z]/.test(password)) {
errors.push('Password must contain lowercase letter');
}
if (!/[0-9]/.test(password)) {
errors.push('Password must contain number');
}
return { valid: errors.length === 0, errors };
}
}
// Unit tests (Jest)
describe('UserValidator', () => {
let validator: UserValidator;
beforeEach(() => {
validator = new UserValidator();
});
describe('validateEmail', () => {
it('should accept valid email', () => {
expect(validator.validateEmail('[email protected]')).toBe(true);
});
it('should reject email without @', () => {
expect(validator.validateEmail('userexample.com')).toBe(false);
});
it('should reject email without domain', () => {
expect(validator.validateEmail('user@')).toBe(false);
});
it('should reject email with spaces', () => {
expect(validator.validateEmail('user @example.com')).toBe(false);
});
});
describe('validatePassword', () => {
it('should accept strong password', () => {
const result = validator.validatePassword('StrongPass123');
expect(result.valid).toBe(true);
expect(result.errors).toHaveLength(0);
});
it('should reject short password', () => {
const result = validator.validatePassword('Short1');
expect(result.valid).toBe(false);
expect(result.errors).toContain('Password must be at least 8 characters');
});
it('should reject password without uppercase', () => {
const result = validator.validatePassword('lowercase123');
expect(result.valid).toBe(false);
expect(result.errors).toContain('Password must contain uppercase letter');
});
it('should return multiple errors for weak password', () => {
const result = validator.validatePassword('weak');
expect(result.valid).toBe(false);
expect(result.errors.length).toBeGreaterThan(1);
});
});
});
Test Doubles: Mocks, Stubs, and Fakes
Test doubles replace real dependencies in tests. Different types serve different purposes.
Types of Test Doubles
Dummy: Passed but never used (fills parameter lists) Stub: Returns canned responses Spy: Records how it was called Mock: Verifies behavior (method calls, arguments) Fake: Working implementation (in-memory database)
Mocking Example (Python)
from unittest.mock import Mock, patch, MagicMock
import pytest
from datetime import datetime
# Code under test
class UserService:
def __init__(self, user_repo, email_service, logger):
self.user_repo = user_repo
self.email_service = email_service
self.logger = logger
def register_user(self, email: str, name: str, password: str):
# Check if user exists
existing = self.user_repo.find_by_email(email)
if existing:
raise ValueError("Email already registered")
# Create user
user = User(email=email, name=name, password=password)
self.user_repo.save(user)
# Send welcome email
self.email_service.send_welcome_email(email, name)
# Log registration
self.logger.info(f"User registered: {email}")
return user
# Tests with mocks
class TestUserService:
@pytest.fixture
def mock_user_repo(self):
return Mock()
@pytest.fixture
def mock_email_service(self):
return Mock()
@pytest.fixture
def mock_logger(self):
return Mock()
@pytest.fixture
def service(self, mock_user_repo, mock_email_service, mock_logger):
return UserService(mock_user_repo, mock_email_service, mock_logger)
def test_register_user_success(self, service, mock_user_repo, mock_email_service):
"""Test successful user registration."""
# Arrange
mock_user_repo.find_by_email.return_value = None
# Act
user = service.register_user('[email protected]', 'Test User', 'password123')
# Assert
assert user.email == '[email protected]'
mock_user_repo.save.assert_called_once()
mock_email_service.send_welcome_email.assert_called_once_with(
'[email protected]', 'Test User'
)
def test_register_user_duplicate_email(self, service, mock_user_repo):
"""Test registration with existing email."""
# Arrange
existing_user = Mock(email='[email protected]')
mock_user_repo.find_by_email.return_value = existing_user
# Act & Assert
with pytest.raises(ValueError, match="already registered"):
service.register_user('[email protected]', 'Test', 'password123')
# Verify save was not called
mock_user_repo.save.assert_not_called()
def test_register_user_logs_registration(self, service, mock_user_repo, mock_logger):
"""Test that registration is logged."""
mock_user_repo.find_by_email.return_value = None
service.register_user('[email protected]', 'Test', 'password123')
mock_logger.info.assert_called_once()
call_args = mock_logger.info.call_args[0][0]
assert '[email protected]' in call_args
Fake Implementation Example
# Fake repository for testing
class FakeUserRepository:
"""In-memory fake repository for testing."""
def __init__(self):
self.users = {}
def save(self, user):
self.users[user.id] = user
def find_by_id(self, user_id):
return self.users.get(user_id)
def find_by_email(self, email):
return next((u for u in self.users.values() if u.email == email), None)
def find_all(self):
return list(self.users.values())
# Test using fake
def test_user_service_with_fake():
"""Test using fake repository instead of mocks."""
fake_repo = FakeUserRepository()
email_service = Mock()
logger = Mock()
service = UserService(fake_repo, email_service, logger)
# Register user
user = service.register_user('[email protected]', 'Test', 'password123')
# Verify user was saved
saved_user = fake_repo.find_by_id(user.id)
assert saved_user is not None
assert saved_user.email == '[email protected]'
# Verify can't register duplicate
with pytest.raises(ValueError):
service.register_user('[email protected]', 'Another', 'password456')
Integration Testing
Integration tests verify that multiple components work together correctly. They test interactions with databases, external APIs, message queues, and other services.
Database Integration Test
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
@pytest.fixture(scope='function')
def test_db():
"""Create test database for each test."""
engine = create_engine('sqlite:///:memory:')
Base.metadata.create_all(engine)
TestingSessionLocal = sessionmaker(bind=engine)
session = TestingSessionLocal()
yield session
session.close()
class TestUserRepository:
"""Integration tests for UserRepository."""
def test_save_and_retrieve_user(self, test_db):
"""Test saving and retrieving user from database."""
repo = SQLUserRepository(test_db)
# Create and save user
user = User(email='[email protected]', name='Test User')
repo.save(user)
# Retrieve user
retrieved = repo.find_by_id(user.id)
assert retrieved is not None
assert retrieved.email == '[email protected]'
assert retrieved.name == 'Test User'
def test_find_by_email(self, test_db):
"""Test finding user by email."""
repo = SQLUserRepository(test_db)
user = User(email='[email protected]', name='Test')
repo.save(user)
found = repo.find_by_email('[email protected]')
assert found is not None
assert found.id == user.id
def test_update_user(self, test_db):
"""Test updating user."""
repo = SQLUserRepository(test_db)
user = User(email='[email protected]', name='Original Name')
repo.save(user)
# Update
user.name = 'Updated Name'
repo.save(user)
# Verify
updated = repo.find_by_id(user.id)
assert updated.name == 'Updated Name'
API Integration Test
from fastapi.testclient import TestClient
import pytest
@pytest.fixture
def client():
"""Create test client."""
from main import app
return TestClient(app)
class TestUserAPI:
"""Integration tests for User API."""
def test_create_user(self, client):
"""Test creating user via API."""
response = client.post('/api/users', json={
'email': '[email protected]',
'name': 'Test User',
'password': 'SecurePass123'
})
assert response.status_code == 201
data = response.json()
assert data['email'] == '[email protected]'
assert 'id' in data
assert 'password' not in data # Password should not be returned
def test_get_user(self, client):
"""Test retrieving user via API."""
# Create user
create_response = client.post('/api/users', json={
'email': '[email protected]',
'name': 'Test User',
'password': 'SecurePass123'
})
user_id = create_response.json()['id']
# Get user
response = client.get(f'/api/users/{user_id}')
assert response.status_code == 200
data = response.json()
assert data['id'] == user_id
assert data['email'] == '[email protected]'
def test_create_user_duplicate_email(self, client):
"""Test creating user with duplicate email."""
# Create first user
client.post('/api/users', json={
'email': '[email protected]',
'name': 'User 1',
'password': 'Pass123'
})
# Try to create duplicate
response = client.post('/api/users', json={
'email': '[email protected]',
'name': 'User 2',
'password': 'Pass456'
})
assert response.status_code == 400
assert 'already exists' in response.json()['detail'].lower()
End-to-End (E2E) Testing
E2E tests verify the entire system from the user’s perspective. They’re slow and brittle but catch integration issues that unit and integration tests miss.
E2E Test Example (Playwright)
import { test, expect } from '@playwright/test';
test.describe('User Registration Flow', () => {
test('should register new user successfully', async ({ page }) => {
// Navigate to registration page
await page.goto('https://example.com/register');
// Fill registration form
await page.fill('[data-testid="email-input"]', '[email protected]');
await page.fill('[data-testid="name-input"]', 'Test User');
await page.fill('[data-testid="password-input"]', 'SecurePass123');
await page.fill('[data-testid="confirm-password-input"]', 'SecurePass123');
// Submit form
await page.click('[data-testid="register-button"]');
// Wait for success message
await expect(page.locator('[data-testid="success-message"]')).toBeVisible();
await expect(page.locator('[data-testid="success-message"]')).toContainText(
'Registration successful'
);
// Verify redirect to dashboard
await expect(page).toHaveURL(/.*\/dashboard/);
});
test('should show validation errors for invalid input', async ({ page }) => {
await page.goto('https://example.com/register');
// Submit empty form
await page.click('[data-testid="register-button"]');
// Check for validation errors
await expect(page.locator('[data-testid="email-error"]')).toBeVisible();
await expect(page.locator('[data-testid="password-error"]')).toBeVisible();
});
test('should prevent duplicate email registration', async ({ page }) => {
await page.goto('https://example.com/register');
// Try to register with existing email
await page.fill('[data-testid="email-input"]', '[email protected]');
await page.fill('[data-testid="name-input"]', 'Test');
await page.fill('[data-testid="password-input"]', 'Pass123');
await page.fill('[data-testid="confirm-password-input"]', 'Pass123');
await page.click('[data-testid="register-button"]');
// Check for error message
await expect(page.locator('[data-testid="error-message"]')).toContainText(
'Email already registered'
);
});
});
Property-Based Testing
Property-based testing generates random inputs to find edge cases you didn’t think of.
from hypothesis import given, strategies as st
import pytest
# Code under test
def reverse_string(s: str) -> str:
return s[::-1]
def is_palindrome(s: str) -> bool:
return s == s[::-1]
# Property-based tests
class TestStringOperations:
@given(st.text())
def test_reverse_twice_returns_original(self, s):
"""Property: reversing twice returns original string."""
assert reverse_string(reverse_string(s)) == s
@given(st.text())
def test_reverse_length_unchanged(self, s):
"""Property: reversing doesn't change length."""
assert len(reverse_string(s)) == len(s)
@given(st.text())
def test_palindrome_equals_reverse(self, s):
"""Property: palindrome equals its reverse."""
if is_palindrome(s):
assert s == reverse_string(s)
@given(st.lists(st.integers()))
def test_sort_idempotent(self, lst):
"""Property: sorting twice gives same result."""
sorted_once = sorted(lst)
sorted_twice = sorted(sorted_once)
assert sorted_once == sorted_twice
@given(st.lists(st.integers(), min_size=1))
def test_max_in_list(self, lst):
"""Property: max element is >= all elements."""
maximum = max(lst)
assert all(maximum >= x for x in lst)
Test Coverage and Mutation Testing
Code Coverage
# Python coverage
pytest --cov=src --cov-report=html tests/
# JavaScript coverage
jest --coverage
# Go coverage
go test -cover ./...
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out
Mutation Testing
Mutation testing changes your code to verify tests catch the changes.
# Install mutpy
# pip install mutpy
# Run mutation testing
# mut.py --target src/ --unit-test tests/ --runner pytest
Best Practices
- Follow AAA pattern - Arrange, Act, Assert
- One assertion per test - Or closely related assertions
- Test behavior, not implementation - Don’t test private methods
- Use descriptive test names -
test_user_cannot_withdraw_more_than_balance - Keep tests independent - No shared state between tests
- Fast feedback - Unit tests should run in seconds
- Don’t test frameworks - Test your code, not Django/Flask/Express
- Use fixtures wisely - Share setup, not state
- Test edge cases - Null, empty, boundary values
- Refactor tests - Tests are code too
Conclusion
Effective testing requires strategy: write many fast unit tests for business logic, some integration tests for component interactions, and few E2E tests for critical user flows. Use test doubles (mocks, stubs, fakes) to isolate units, follow the AAA pattern for clarity, and test behavior rather than implementation. Aim for meaningful coverage that catches real bugs, not arbitrary percentage targets. Property-based testing finds edge cases, mutation testing verifies test quality, and continuous refactoring keeps tests maintainable. Remember: tests are an investment in confidence, not a checkbox to complete.
Resources
- Test-Driven Development by Kent Beck
- Working Effectively with Legacy Code by Michael Feathers
- pytest Documentation
- Jest Documentation
- Playwright Documentation
- Hypothesis (Property-Based Testing)
- The Practical Test Pyramid by Martin Fowler
Comments