Mutation Testing: Verify Your Tests Actually Work

Introduction

You can have 100% code coverage but still have bad tests. Mutation testing verifies test quality by introducing small changes (mutations) to your code and checking if tests catch them.

How Mutation Testing Works

┌─────────────────────────────────────────────────────────────┐
│              Mutation Testing Process                          │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Original Code:                                            │
│  ┌─────────────────────────────────────────┐               │
│  │  function add(a, b) {                  │               │
│  │    return a + b;                       │               │
│  │  }                                     │               │
│  └─────────────────────────────────────────┘               │
│                         │                                    │
│                         ▼                                    │
│  Mutation: Change + to -                                    │
│  ┌─────────────────────────────────────────┐               │
│  │  function add(a, b) {                  │               │
│  │    return a - b;    ← MUTANT            │               │
│  │  }                                     │               │
│  └─────────────────────────────────────────┘               │
│                         │                                    │
│                         ▼                                    │
│  Run Tests:                                                │
│  ┌─────────────────────────────────────────┐               │
│  │  add(2, 2) should return 4              │               │
│  │                                        │               │
│  │  Mutant survives? ❌ (test failed)   │               │
│  │  Tests caught it! ✓                     │               │
│  └─────────────────────────────────────────┘               │
│                                                             │
│  Mutation Score = Caught / Total Mutants                    │
│                                                             │
└─────────────────────────────────────────────────────────────┘

JavaScript/TypeScript: Stryker

Setup

npm install -D @stryker-mutator/core
npx stryker init

Configuration

// stryker.conf.json
{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "mutator": "typescript",
  "packageManager": "npm",
  "reporters": ["html", "clear-text", "dashboard"],
  "buildCommand": "npm run build",
  "testRunner": "command",
  "commandRunner": {
    "command": "npm test"
  },
  "mutate": [
    "src/**/*.ts",
    "!src/**/*.spec.ts"
  ],
  "thresholds": {
    "high": 80,
    "low": 70,
    "break": 60
  }
}

Running

# Run mutation testing
npx stryker run

# Output:
# Mutant killed: 45/50 (90%)
# ┌───────────────────────────────────────┐
# │ File          │ Mutation score      │
# │ ──────────────│───────────────────── │
# │ math.ts       │ 95%                  │
# │ string.ts     │ 85%                  │
# │ utils.ts      │ 75% ⚠️               │
# └───────────────────────────────────────┘

Java: PIT

Setup

<!-- pom.xml -->
<build>
  <plugins>
    <plugin>
      <groupId>org.pitest</groupId>
      <artifactId>pitest-maven</artifactId>
      <version>1.15.0</version>
    </plugin>
  </plugins>
</build>

Running

mvn org.pitest:pitest-maven:mutationCoverage

Results

================================================================================
>> mutation coverage report
================================================================================

Classes : 85% (34/40)
Methods : 80% (120/150)
Mutations: 75% (150/200)

UNCOVERED MUTATIONS:
================================================================================
com.example.Utils.java:
  Line 45: replaced + with -                              SURVIVED
  Line 67: removed conditional - omitted <               SURVIVED

What Gets Mutated

mutations:
  - "Arithmetic: + → -, * → /, etc."
  - "Boolean: true → false, && → ||"
  - "Comparison: == → !=, < → >="
  - "Conditional: removed if/else branches"
  - "String: '' → '', .length → 0"
  - "Array: [0] → arr.first()"

Interpretation

# Mutation score interpretation
scores:
  above_80:
    label: "Excellent"
    description: "Tests are catching most bugs"
    
  70_80:
    label: "Good"
    description: "Room for improvement"
    
  60_70:
    label: "Warning"
    description: "Some bugs may slip through"
    
  below_60:
    label: "Critical"
    description: "Tests not effective"

Key Takeaways

• Mutation testing - Verifies test quality, not just coverage
• Stryker - JavaScript/TypeScript
• PIT - Java
• Goal - Catch as many mutants as possible

External Resources

• Stryker Documentation
• PIT Mutation Testing
• Mutation Testing Guide