Log Arithmetic

Logarithmic Arithmetic in Python

2019-06-23

When working with probabilities or likelihoods, the values are often extremely small (close to zero). In Python, numbers smaller than about 1e-300 (for 64-bit floats) are rounded to zero, leading to loss of precision. Multiplying many small fractional numbers can quickly underflow to zero.

Solution: Work in the log domain. By converting numbers to their logarithms, you avoid underflow and maintain numerical stability.

Example

import numpy as np
from scipy.special import logsumexp

a, b = 1e-323, 1e-324
print(a, b)
# Output: 1e-323 0.0

# Logarithmic calculations
a = np.log(1.25) - 500 * np.log(10)  # log(1.25e-500)
b = np.log(3.54) - 500 * np.log(10)  # log(3.54e-500)
print(a, b, logsumexp([a, b]))
# Output: -1151.0694029457086 -1150.0284197698772 -1149.7260160856001

# Convert back from log domain
result = np.exp(-1149.7260160856001 + 500 * np.log(10))
print(result)
# Output: 4.789999999999495

Key Points

  • Underflow: Multiplying many small numbers can result in zero due to floating-point underflow.
  • Log Domain: Use logarithms to represent small numbers and perform addition instead of multiplication.
  • logsumexp: Use scipy.special.logsumexp for numerically stable log-domain summation.
  • Precision: Working in the log domain preserves precision for very small values.

References