Introspection and Reflection in Python: Examining and Modifying Program Structure at Runtime
Python’s introspection and reflection capabilities are among its most powerful features. They allow programs to examine their own structure, understand how objects work, and even modify behavior at runtime. These capabilities enable sophisticated patterns like dependency injection, serialization frameworks, testing tools, and plugin systems.
Yet many developers use these features without fully understanding them. This guide explores introspection and reflection, showing you how to leverage them effectively in real-world scenarios.
Table of Contents
- Core Concepts
- Built-in Introspection Functions
- The inspect Module
- Practical Use Cases
- Reflection: Modifying Program Structure
- Best Practices and Pitfalls
- Conclusion
Core Concepts
What is Introspection?
Introspection is the ability to examine objects, classes, functions, and modules at runtime. It answers questions like:
- What type is this object?
- What attributes does this object have?
- What methods can I call on this object?
- What are the parameters of this function?
- What is the source code of this function?
# Simple introspection examples
class Dog:
def bark(self):
return "Woof!"
dog = Dog()
# What type is this?
print(type(dog)) # <class '__main__.Dog'>
# What attributes does it have?
print(dir(dog)) # ['bark', '__class__', '__delattr__', ...]
# Can I call bark?
print(callable(dog.bark)) # True
What is Reflection?
Reflection is the ability to modify program structure at runtime. It answers questions like:
- Can I add a new attribute to an object?
- Can I call a method by name?
- Can I modify a class definition?
- Can I create new classes dynamically?
# Simple reflection examples
class Dog:
pass
dog = Dog()
# Add an attribute dynamically
dog.name = "Buddy"
print(dog.name) # Buddy
# Call a method by name
def bark(self):
return "Woof!"
Dog.bark = bark
print(dog.bark()) # Woof!
# Modify the class
Dog.species = "Canis familiaris"
print(Dog.species) # Canis familiaris
Introspection vs Reflection
| Aspect | Introspection | Reflection |
|---|---|---|
| Purpose | Examine program structure | Modify program structure |
| Direction | Read-only | Read and write |
| Examples | type(), dir(), inspect |
setattr(), delattr(), dynamic class creation |
| Use Cases | Debugging, documentation, testing | Dependency injection, serialization, plugins |
Built-in Introspection Functions
Python provides several built-in functions for introspection.
type()
Returns the type of an object:
# Basic types
print(type(42)) # <class 'int'>
print(type("hello")) # <class 'str'>
print(type([1, 2, 3])) # <class 'list'>
print(type({'a': 1})) # <class 'dict'>
# Custom classes
class Dog:
pass
dog = Dog()
print(type(dog)) # <class '__main__.Dog'>
print(type(Dog)) # <class 'type'>
# Check type relationships
print(type(dog) is Dog) # True
print(isinstance(dog, Dog)) # True
isinstance() and issubclass()
Check type relationships:
class Animal:
pass
class Dog(Animal):
pass
dog = Dog()
# Check if object is instance of class
print(isinstance(dog, Dog)) # True
print(isinstance(dog, Animal)) # True (inheritance)
print(isinstance(dog, str)) # False
# Check class relationships
print(issubclass(Dog, Animal)) # True
print(issubclass(Dog, Dog)) # True (a class is subclass of itself)
print(issubclass(str, Animal)) # False
dir()
Lists attributes and methods of an object:
class Dog:
species = "Canis familiaris"
def __init__(self, name):
self.name = name
def bark(self):
return "Woof!"
dog = Dog("Buddy")
# Get all attributes and methods
attributes = dir(dog)
print(attributes)
# ['__class__', '__delattr__', ..., 'bark', 'name', 'species']
# Filter to show only custom attributes
custom = [attr for attr in dir(dog) if not attr.startswith('_')]
print(custom) # ['bark', 'name', 'species']
getattr(), setattr(), hasattr(), delattr()
Access and modify attributes dynamically:
class Config:
debug = True
timeout = 30
config = Config()
# Get attribute
print(getattr(config, 'debug')) # True
print(getattr(config, 'missing', 'default')) # default
# Check if attribute exists
print(hasattr(config, 'debug')) # True
print(hasattr(config, 'missing')) # False
# Set attribute
setattr(config, 'debug', False)
print(config.debug) # False
# Set new attribute
setattr(config, 'new_attr', 'value')
print(config.new_attr) # value
# Delete attribute
delattr(config, 'new_attr')
print(hasattr(config, 'new_attr')) # False
callable()
Check if an object can be called:
def my_function():
pass
class MyClass:
def __call__(self):
return "Called!"
# Functions are callable
print(callable(my_function)) # True
# Classes are callable (they create instances)
print(callable(MyClass)) # True
# Instances with __call__ are callable
obj = MyClass()
print(callable(obj)) # True
# Regular objects are not callable
print(callable(42)) # False
print(callable("string")) # False
The inspect Module
The inspect module provides advanced introspection capabilities.
Inspecting Functions
import inspect
def greet(name, greeting="Hello"):
"""Greet someone with a custom greeting"""
return f"{greeting}, {name}!"
# Get function signature
sig = inspect.signature(greet)
print(sig) # (name, greeting='Hello')
# Get parameters
for param_name, param in sig.parameters.items():
print(f"{param_name}: {param.default}")
# name: <class 'inspect._empty'>
# greeting: Hello
# Get function source code
print(inspect.getsource(greet))
# def greet(name, greeting="Hello"):
# """Greet someone with a custom greeting"""
# return f"{greeting}, {name}!"
# Get docstring
print(inspect.getdoc(greet))
# Greet someone with a custom greeting
# Check if it's a function
print(inspect.isfunction(greet)) # True
Inspecting Classes
import inspect
class Animal:
def move(self):
pass
class Dog(Animal):
def __init__(self, name):
self.name = name
def bark(self):
return "Woof!"
@staticmethod
def species():
return "Canis familiaris"
# Get class members
members = inspect.getmembers(Dog)
print(members) # List of (name, value) tuples
# Get methods
methods = inspect.getmembers(Dog, predicate=inspect.ismethod)
print(methods)
# Get class hierarchy
print(inspect.getmro(Dog)) # (Dog, Animal, object)
# Check method types
print(inspect.ismethod(Dog().bark)) # True
print(inspect.isfunction(Dog.bark)) # True
print(inspect.isbuiltin(len)) # True
Inspecting Call Stacks
import inspect
def level_3():
# Get the current call stack
stack = inspect.stack()
for frame_info in stack:
print(f"Function: {frame_info.function}")
print(f"File: {frame_info.filename}")
print(f"Line: {frame_info.lineno}")
print()
def level_2():
level_3()
def level_1():
level_2()
level_1()
# Output shows the call stack from level_1 → level_2 → level_3
Getting Source Code
import inspect
class Calculator:
def add(self, a, b):
"""Add two numbers"""
return a + b
# Get source of a method
source = inspect.getsource(Calculator.add)
print(source)
# def add(self, a, b):
# """Add two numbers"""
# return a + b
# Get source file and line number
print(inspect.getsourcefile(Calculator)) # /path/to/file.py
print(inspect.getsourcelines(Calculator)[1]) # Line number where class starts
Practical Use Cases
Use Case 1: Automatic Serialization
import json
from typing import Any
class Serializable:
"""Base class that provides automatic JSON serialization"""
def to_dict(self) -> dict:
"""Convert object to dictionary using introspection"""
result = {}
for attr_name in dir(self):
# Skip private attributes and methods
if attr_name.startswith('_'):
continue
attr_value = getattr(self, attr_name)
# Skip methods
if callable(attr_value):
continue
result[attr_name] = attr_value
return result
def to_json(self) -> str:
"""Convert object to JSON string"""
return json.dumps(self.to_dict())
@classmethod
def from_dict(cls, data: dict) -> 'Serializable':
"""Create object from dictionary"""
obj = cls.__new__(cls)
for key, value in data.items():
setattr(obj, key, value)
return obj
# Usage
class User(Serializable):
def __init__(self, name: str, email: str, age: int):
self.name = name
self.email = email
self.age = age
user = User("Alice", "[email protected]", 30)
# Serialize
print(user.to_json())
# {"name": "Alice", "email": "[email protected]", "age": 30}
# Deserialize
user_dict = {"name": "Bob", "email": "[email protected]", "age": 25}
user2 = User.from_dict(user_dict)
print(user2.name) # Bob
Use Case 2: Dependency Injection
import inspect
from typing import Any, Dict, Type
class Container:
"""Simple dependency injection container"""
def __init__(self):
self.services: Dict[str, Any] = {}
def register(self, name: str, service: Any) -> None:
"""Register a service"""
self.services[name] = service
def resolve(self, cls: Type) -> Any:
"""Resolve a class by injecting dependencies"""
# Get the __init__ signature
sig = inspect.signature(cls.__init__)
# Get parameters (skip 'self')
params = list(sig.parameters.keys())[1:]
# Resolve each parameter
kwargs = {}
for param in params:
if param in self.services:
kwargs[param] = self.services[param]
else:
raise ValueError(f"Cannot resolve parameter: {param}")
# Create instance with resolved dependencies
return cls(**kwargs)
# Define services
class Database:
def query(self, sql):
return f"Executing: {sql}"
class Logger:
def log(self, message):
print(f"[LOG] {message}")
# Define a class that depends on services
class UserService:
def __init__(self, database: Database, logger: Logger):
self.database = database
self.logger = logger
def get_user(self, user_id):
self.logger.log(f"Getting user {user_id}")
return self.database.query(f"SELECT * FROM users WHERE id = {user_id}")
# Set up container
container = Container()
container.register('database', Database())
container.register('logger', Logger())
# Resolve and use
user_service = container.resolve(UserService)
print(user_service.get_user(1))
# [LOG] Getting user 1
# Executing: SELECT * FROM users WHERE id = 1
Use Case 3: Plugin System
import inspect
import importlib
from pathlib import Path
from typing import Dict, Type
class PluginRegistry:
"""Registry for dynamically loaded plugins"""
def __init__(self):
self.plugins: Dict[str, Type] = {}
def register(self, name: str, plugin_class: Type) -> None:
"""Register a plugin"""
self.plugins[name] = plugin_class
def load_plugins_from_directory(self, directory: str) -> None:
"""Dynamically load plugins from a directory"""
path = Path(directory)
for file in path.glob("*.py"):
if file.name.startswith("_"):
continue
# Import the module
module_name = file.stem
spec = importlib.util.spec_from_file_location(module_name, file)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# Find plugin classes
for name, obj in inspect.getmembers(module):
if inspect.isclass(obj) and hasattr(obj, 'plugin_name'):
self.register(obj.plugin_name, obj)
def get_plugin(self, name: str) -> Type:
"""Get a plugin by name"""
return self.plugins.get(name)
def list_plugins(self) -> list:
"""List all registered plugins"""
return list(self.plugins.keys())
# Define plugin base class
class Plugin:
plugin_name = None
def execute(self):
raise NotImplementedError
# Example plugins
class ImagePlugin(Plugin):
plugin_name = "image"
def execute(self):
return "Processing image"
class VideoPlugin(Plugin):
plugin_name = "video"
def execute(self):
return "Processing video"
# Use the registry
registry = PluginRegistry()
registry.register("image", ImagePlugin)
registry.register("video", VideoPlugin)
print(registry.list_plugins()) # ['image', 'video']
# Execute a plugin
plugin = registry.get_plugin("image")()
print(plugin.execute()) # Processing image
Use Case 4: Testing Framework
import inspect
from typing import Callable, List
class TestRunner:
"""Simple test runner using introspection"""
def __init__(self):
self.tests: List[Callable] = []
self.passed = 0
self.failed = 0
def discover_tests(self, test_class: type) -> None:
"""Discover test methods in a class"""
for name, method in inspect.getmembers(test_class):
# Find methods that start with 'test_'
if name.startswith('test_') and inspect.isfunction(method):
self.tests.append((name, method))
def run_tests(self, test_class: type) -> None:
"""Run all discovered tests"""
self.discover_tests(test_class)
instance = test_class()
print(f"Running {len(self.tests)} tests...\n")
for test_name, test_method in self.tests:
try:
# Call the test method
test_method(instance)
print(f"✓ {test_name}")
self.passed += 1
except AssertionError as e:
print(f"✗ {test_name}: {e}")
self.failed += 1
except Exception as e:
print(f"✗ {test_name}: {type(e).__name__}: {e}")
self.failed += 1
print(f"\n{self.passed} passed, {self.failed} failed")
# Define tests
class TestCalculator:
def test_addition(self):
assert 2 + 2 == 4
def test_subtraction(self):
assert 5 - 3 == 2
def test_multiplication(self):
assert 3 * 4 == 12
def test_division_fails(self):
assert 10 / 2 == 5
# Run tests
runner = TestRunner()
runner.run_tests(TestCalculator)
# Output:
# Running 4 tests...
# ✓ test_addition
# ✓ test_subtraction
# ✓ test_multiplication
# ✓ test_division_fails
# 4 passed, 0 failed
Use Case 5: Debugging and Introspection Tools
import inspect
from typing import Any
class DebugHelper:
"""Helper for debugging objects"""
@staticmethod
def inspect_object(obj: Any) -> None:
"""Print detailed information about an object"""
print(f"Object: {obj}")
print(f"Type: {type(obj)}")
print(f"Module: {type(obj).__module__}")
print()
# Get attributes
print("Attributes:")
for attr_name in dir(obj):
if not attr_name.startswith('_'):
try:
attr_value = getattr(obj, attr_name)
if not callable(attr_value):
print(f" {attr_name}: {attr_value}")
except:
pass
print()
# Get methods
print("Methods:")
for attr_name in dir(obj):
if not attr_name.startswith('_'):
try:
attr_value = getattr(obj, attr_name)
if callable(attr_value):
sig = inspect.signature(attr_value)
print(f" {attr_name}{sig}")
except:
pass
@staticmethod
def get_object_tree(obj: Any, max_depth: int = 2, current_depth: int = 0) -> None:
"""Print object hierarchy"""
indent = " " * current_depth
print(f"{indent}{type(obj).__name__}")
if current_depth >= max_depth:
return
for attr_name in dir(obj):
if not attr_name.startswith('_'):
try:
attr_value = getattr(obj, attr_name)
if not callable(attr_value):
print(f"{indent} {attr_name}: {type(attr_value).__name__}")
except:
pass
# Usage
class Person:
def __init__(self, name, age):
self.name = name
self.age = age
def greet(self):
return f"Hello, I'm {self.name}"
person = Person("Alice", 30)
DebugHelper.inspect_object(person)
Reflection: Modifying Program Structure
Reflection allows you to modify program structure at runtime.
Dynamic Attribute Assignment
class DynamicObject:
pass
obj = DynamicObject()
# Add attributes dynamically
obj.name = "Alice"
obj.age = 30
obj.email = "[email protected]"
print(obj.name) # Alice
print(obj.age) # 30
# Add methods dynamically
def greet(self):
return f"Hello, I'm {self.name}"
DynamicObject.greet = greet
print(obj.greet()) # Hello, I'm Alice
Dynamic Class Creation
# Create a class dynamically
def __init__(self, name, age):
self.name = name
self.age = age
def greet(self):
return f"Hello, I'm {self.name}"
Person = type('Person', (object,), {
'__init__': __init__,
'greet': greet,
})
person = Person("Bob", 25)
print(person.greet()) # Hello, I'm Bob
Monkey Patching
# Original class
class Calculator:
def add(self, a, b):
return a + b
# Monkey patch to add new functionality
original_add = Calculator.add
def add_with_logging(self, a, b):
result = original_add(self, a, b)
print(f"Added {a} + {b} = {result}")
return result
Calculator.add = add_with_logging
calc = Calculator()
calc.add(2, 3) # Added 2 + 3 = 5
Modifying Class Hierarchy
class Animal:
def move(self):
return "Moving"
class Dog:
def bark(self):
return "Woof!"
# Add Animal as a base class to Dog
Dog.__bases__ = (Animal,)
dog = Dog()
print(dog.move()) # Moving
print(dog.bark()) # Woof!
Best Practices and Pitfalls
Best Practice 1: Use Introspection for Frameworks
Introspection is perfect for frameworks that need to understand user code:
# Good: Framework using introspection
class APIEndpoint:
def __init__(self, handler_class):
self.handler = handler_class()
self.methods = self._discover_methods()
def _discover_methods(self):
"""Discover HTTP handler methods"""
methods = {}
for name, method in inspect.getmembers(self.handler):
if name.startswith('handle_'):
http_method = name.replace('handle_', '').upper()
methods[http_method] = method
return methods
Best Practice 2: Avoid Reflection When Possible
Reflection is powerful but can make code hard to understand:
# Bad: Unnecessary reflection
def call_method(obj, method_name):
method = getattr(obj, method_name)
return method()
# Good: Direct method call
def call_method(obj):
return obj.method()
Pitfall 1: Performance Overhead
Introspection and reflection have performance costs:
import timeit
class MyClass:
def method(self):
return 42
obj = MyClass()
# Direct call
direct_time = timeit.timeit(lambda: obj.method(), number=1000000)
# Using getattr
getattr_time = timeit.timeit(
lambda: getattr(obj, 'method')(),
number=1000000
)
print(f"Direct: {direct_time:.4f}s")
print(f"getattr: {getattr_time:.4f}s")
# getattr is typically 2-3x slower
Pitfall 2: Breaking Encapsulation
Reflection can bypass intended access restrictions:
class BankAccount:
def __init__(self, balance):
self._balance = balance # Private attribute
def withdraw(self, amount):
if amount > self._balance:
raise ValueError("Insufficient funds")
self._balance -= amount
account = BankAccount(100)
# Reflection bypasses validation
setattr(account, '_balance', 1000000) # Oops!
print(account._balance) # 1000000
Pitfall 3: Fragile Code
Reflection makes code fragile to refactoring:
# Fragile: Depends on method name as string
def call_handler(obj, action):
method_name = f"handle_{action}"
method = getattr(obj, method_name, None)
if method:
return method()
# If you rename handle_save to handle_persist, this breaks
Best Practice 3: Document Introspection Usage
Make it clear when code uses introspection:
def serialize_object(obj):
"""
Serialize an object to a dictionary using introspection.
This function uses introspection to discover all public attributes
of the object and convert them to a dictionary.
Args:
obj: Any object with public attributes
Returns:
Dictionary of attribute names to values
"""
result = {}
for attr_name in dir(obj):
if not attr_name.startswith('_'):
attr_value = getattr(obj, attr_name)
if not callable(attr_value):
result[attr_name] = attr_value
return result
Conclusion
Introspection and reflection are powerful Python features that enable sophisticated programming patterns. Introspection lets you examine program structure at runtime, while reflection lets you modify it.
Key takeaways:
- Introspection examines objects, classes, and functions at runtime
- Reflection modifies program structure at runtime
- Built-in functions like
type(),dir(),getattr()provide basic introspection - The inspect module provides advanced introspection capabilities
- Use introspection for frameworks, debugging, and testing
- Use reflection for dependency injection, plugins, and dynamic behavior
- Avoid reflection when simpler alternatives exist
- Document when code uses introspection or reflection
- Consider performance implications of introspection-heavy code
With these tools in your toolkit, you can build flexible, powerful Python applications that understand and adapt to their own structure.
Comments