Introduction
Unit tests form the foundation of software testing, but comprehensive quality requires much more. Modern applications need multiple testing layers, from fast unit tests to slow but thorough end-to-end tests. This guide explores testing strategies beyond unit testing that help build reliable software.
The Testing Pyramid
Understanding the Pyramid
/\
/E2E\ Few, Slow, Expensive
/----\
/Integration\ Medium frequency
/------ ----\
/ Unit Tests \ Many, Fast, Cheap
/----------------\
Balancing Tests
- Many unit tests: Fast feedback, catch bugs early
- Fewer integration tests: Test component interactions
- Minimal E2E tests: Critical user journeys only
Integration Testing
What It Tests
Integration tests verify that components work together:
- Database interactions
- API integrations
- Service communication
- External dependencies
Database Testing
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
@pytest.fixture
def test_db():
engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(engine)
yield sessionmaker(bind=engine)()
Base.metadata.drop_all(engine)
def test_user_creation(test_db):
user = User(name="John", email="[email protected]")
test_db.add(user)
test_db.commit()
retrieved = test_db.query(User).first()
assert retrieved.name == "John"
API Testing
import pytest
from fastapi.testclient import TestClient
@pytest.fixture
def client():
from main import app
return TestClient(app)
def test_create_order(client, test_db):
response = client.post("/orders", json={
"items": [{"product_id": 1, "quantity": 2}],
"customer_id": 1
})
assert response.status_code == 201
data = response.json()
assert data["id"] is not None
assert len(data["items"]) == 1
Service Integration
def test_order_notification_integration():
# Set up mock notification service
with mock.patch('notifications.send') as mock_send:
order_service = OrderService(notification_service)
order = order_service.create_order(VALID_ORDER)
mock_send.assert_called_once()
call_args = mock_send.call_args
assert "order_id" in call_args[0][0]
End-to-End Testing
When to Use E2E
E2E tests verify complete user flows:
- Critical user journeys
- Payment flows
- Authentication flows
- Core business processes
Playwright Example
from playwright.sync_api import sync_playwright
def test_user_checkout():
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
# Navigate to store
page.goto("https://store.example.com")
# Browse products
page.click(".product:first-child")
# Add to cart
page.click("#add-to-cart")
# Checkout
page.click("#checkout")
page.fill("#email", "[email protected]")
page.fill("#card", "4242424242424242")
page.click("#pay")
# Verify success
assert "Thank you" in page.content()
browser.close()
Best Practices
- Test critical paths only
- Run in parallel when possible
- Use realistic data
- Clean up after tests
Property-Based Testing
Concept
Instead of testing specific inputs, test properties that should hold for all inputs:
from hypothesis import given, strategies as st
@given(st.lists(st.integers(min_value=1, max_value=100)))
def test_sorting_properties(nums):
sorted_nums = sorted(nums)
# Property 1: Same length
assert len(sorted_nums) == len(nums)
# Property 2: Sorted order
assert all(sorted_nums[i] <= sorted_nums[i+1]
for i in range(len(sorted_nums)-1))
# Property 3: Same elements
assert sorted_nums == sorted(nums)
Use Cases
- Cryptographic operations
- Data transformations
- Serialization/deserialization
- Algorithm verification
Python Libraries
- Hypothesis: Most popular property testing
- FAST: Fuzzing for property testing
- AutoPyTest: AI-assisted property discovery
Contract Testing
Purpose
Ensure services can communicate:
- Frontend-backend compatibility
- Microservice interfaces
- API versions
- Provider-consumer agreements
Pact Example
import pytest
from pact import Consumer, Provider
@pytest.fixture
def pact():
return Consumer('web-app').has_pact_with(Provider('user-service'))
def test_get_user(pact):
pact.given('user exists').upon_receiving(
'a request for user'
).with_request(
method='GET',
path='/users/1'
).will_respond_with(
status=200,
body={
'id': 1,
'name': 'John'
}
)
with pact:
response = requests.get('http://localhost:8000/users/1')
assert response.json()['name'] == 'John'
Chaos Engineering
Purpose
Test system resilience by deliberately introducing failures:
- Network failures
- Service downtime
- Resource exhaustion
- Latency injection
Tools
- Chaos Monkey: Netflix’s chaos tool
- Litmus: Kubernetes chaos
- Gremlin: Commercial chaos platform
Example
from chaos import experiment
@experiment
def test_payment_timeout():
# Inject 5s delay into payment service
with chaos.inject().delay("payment-service", seconds=5):
response = client.post("/checkout")
# System should handle gracefully
assert response.status_code in [200, 503]
if response.status_code == 200:
# Should handle with timeout
assert response.elapsed < 10
AI-Assisted Testing
Current Capabilities
AI helps with:
- Test generation from code
- Test maintenance
- Bug detection
- Flaky test identification
Tools
- Diffblue: Unit test generation
- Kite: Python test assistance
- Copilot: Code and test suggestions
Human + AI
Best results combine AI efficiency with human judgment:
- AI generates initial tests
- Humans review and refine
- Humans add edge cases
- Humans verify critical assertions
Testing Strategy Framework
By Application Type
Web Applications
- Unit: Business logic
- Integration: API endpoints
- E2E: Checkout, auth, search
APIs
- Contract: All endpoints
- Integration: Database, external services
- Performance: Load testing
Data Pipelines
- Unit: Transform functions
- Integration: Pipeline steps
- Data quality: Schema, completeness
By Risk Level
| Risk | Testing Level | Investment |
|---|---|---|
| High | Full coverage | High |
| Medium | Key paths | Medium |
| Low | Smoke tests | Low |
Measuring Test Quality
Metrics
- Coverage: % of code tested
- Mutation score: Are tests catching bugs?
- Flakiness: Reliability over time
- Speed: How fast do tests run?
Tools
- Coverage.py: Python coverage
- Mutmut: Mutation testing
- TestRail: Test management
Conclusion
Comprehensive testing requires multiple layers working together. Unit tests provide fast feedback, integration tests verify component collaboration, and E2E tests ensure critical user journeys work. Property-based testing and chaos engineering add robustness. Build your testing strategy based on your application’s risk profile and user expectations.
Comments