Network Troubleshooting: Bandwidth Testing and Latency Diagnostics

Introduction

Network performance issues are among the most frequent causes of application slowdowns and outages. Whether users report sluggish file transfers, video conferencing glitches, or API timeouts, the root cause often lies somewhere in the network path. Without systematic troubleshooting methodologies, administrators waste hours chasing phantom issues.

This guide provides a structured approach to diagnosing network problems, focusing on the fundamental metrics that matter: bandwidth, latency, packet loss, and jitter. You’ll learn to use industry-standard tools, interpret their results correctly, and follow diagnostic workflows that progress from simple checks to advanced analysis.

What You’ll Learn:

• Identifying common network performance problems and their symptoms
• Measuring bandwidth with iperf3 and speedtest-cli
• Diagnosing latency issues using ping, traceroute, and MTR
• Understanding the relationship between bandwidth, latency, packet loss, and jitter
• Building practical troubleshooting workflows
• Differentiating between local, ISP, and application-level issues

Understanding Network Performance Metrics

Before diving into tools, you need to understand the four key metrics that define network performance:

Bandwidth

Bandwidth represents the maximum data transfer capacity of a network connection, typically measured in bits per second (bps). It’s often confused with speed, but bandwidth is actually about capacity, not how fast data travels.

Bandwidth: 1 Gbps = 1,000,000,000 bits per second
Throughput: Actual data transfer rate achieved in practice

Key Distinction: A 1 Gbps link doesn’t guarantee 1 Gbps throughput. Real-world performance depends on network conditions, protocol overhead, and endpoint capabilities.

Latency

Latency measures the time it takes for a data packet to travel from source to destination, typically measured in milliseconds (ms). This is the “ping time” you see in games and network utilities.

Latency = Transmission Time + Propagation Time + Processing Time

Typical Latencies:

Packet Loss

Packet loss occurs when data packets fail to reach their destination. Even small amounts can significantly impact performance, especially for real-time applications.

Acceptable Thresholds:

• Voice/Video: < 1% packet loss
• General Networking: < 0.1% packet loss
• Storage/Backup: < 0.01% packet loss

Jitter

Jitter measures the variation in latency over time. High jitter causes inconsistent performance, leading to buffer underruns in streaming and retransmissions in TCP.

Jitter Guidelines:

• < 20 ms: Excellent for real-time applications
• 20-50 ms: Acceptable for most use cases

• 50 ms: Problematic for voice/video

Common Network Problems and Their Indicators

Understanding symptom patterns helps you narrow down the root cause quickly:

Bandwidth-Related Issues

Symptoms:

• Slow file transfers that take longer than expected
• Multiple users experiencing slowdowns simultaneously
• Performance degradation during peak hours
• Speed test results below subscribed rate

Common Causes:

• ISP throttling or congestion
• Network link saturation
• Inadequate bandwidth for user count
• QoS policies limiting traffic

Latency-Related Issues

Symptoms:

• High ping times to local servers
• Applications feel “laggy” despite good speeds
• Slow DNS resolution
• Interactive applications feel unresponsive

Common Causes:

• Distance to remote servers
• Network congestion at hops
• Inefficient routing
• Underpowered network equipment

Packet Loss Issues

Symptoms:

• Intermittent connection drops
• Retransmissions visible in connection logs
• Corrupted downloads requiring retries
• VoIP calls with audio gaps

Common Causes:

• Faulty network cables or connectors
• Overloaded network equipment
• ISP network issues
• Wireless interference (WiFi)

DNS Resolution Problems

Symptoms:

• Websites load slowly despite fast connection
• Some domains fail to resolve
• Inconsistent domain resolution times
• “Server not found” errors

Common Causes:

• Slow or failing DNS servers
• DNS cache corruption
• ISP DNS outages
• Firewall DNS filtering

Bandwidth Testing Tools and Techniques

Using iperf3 for Network Throughput Testing

iperf3 is the industry standard for measuring network throughput. It supports both TCP and UDP testing, with configurable buffer sizes and parallel streams.

Installation:

# Ubuntu/Debian
sudo apt install iperf3

# macOS
brew install iperf3

# CentOS/RHEL
sudo yum install iperf3

Basic TCP Test (Server-Side Setup):

# On the server (receives traffic)
iperf3 -s -p 5201

# Server with more options
iperf3 -s -p 5201 -i 1 -f M

Basic TCP Test (Client-Side):

# Connect to server and run test
iperf3 -c 192.168.1.100 -p 5201

# With results in Mbps
iperf3 -c 192.168.1.100 -p 5201 -f M

# Run multiple tests with parallel streams
iperf3 -c 192.168.1.100 -p 5201 -P 4 -f M

Sample Output:

Connecting to host 192.168.1.100, port 5201
[  4] local 192.168.1.50 port 54321 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Retr
[  4]   0.00-1.00   sec   112 MBytes   112 MBytes/sec    0
[  4]   1.00-2.00   sec   118 MBytes   118 MBytes/sec    0
[  4]   2.00-3.00   sec   117 MBytes   117 MBytes/sec    0
[  4]   3.00-4.00   sec   118 MBytes   118 MBytes/sec    0
[  4]   4.00-5.00   sec   118 MBytes   118 MBytes/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  4]   0.00-5.00   sec   583 MBytes   116 MBytes/sec    0

UDP Testing for Jitter and Packet Loss:

# Server
iperf3 -s -p 5201

# Client with UDP (default 1 Gbps)
iperf3 -c 192.168.1.100 -p 5201 -u -b 100M

Interpreting iperf3 Results:

• Bitrate: Actual throughput achieved (compare to expected bandwidth)
• Retr: Number of TCP retransmissions (indicates packet loss or congestion)
• Jitter: Variation in latency (UDP tests only)
• Lost/Total Datagrams: Packet loss percentage (UDP tests only)

Using speedtest-cli for Internet Speed Testing

For quick internet speed tests against public servers:

Installation:

# pip
pip install speedtest-cli

# Ubuntu/Debian
sudo apt install speedtest-cli

# macOS
brew install speedtest-cli

Basic Usage:

# Simple test
speedtest-cli

# Verbose output
speedtest-cli --verbose

# Minimal output
speedtest-cli --simple

# Specify server by ID
speedtest-cli --server 1234

Sample Output:

Testing download speed................................................................................
Download: 94.23 Mbit/s
Testing upload speed................................................................................
Upload: 45.67 Mbit/s
Ping: 12.34 ms
Server: ISP Location (City, ST)

Using speedtest-cli in Scripts:

#!/bin/bash
# Get JSON output for parsing
result=$(speedtest-cli --json)

download=$(echo "$result" | jq -r '.download.bandwidth')
upload=$(echo "$result" | jq -r '.upload.bandwidth')
ping=$(echo "$result" | jq -r '.ping.latency')

echo "Download: $((download / 125000)) Mbps"
echo "Upload: $((upload / 125000)) Mbps"
echo "Ping: $ping ms"

Testing Local Network Bandwidth

For testing internal network segments:

# Using iperf3 between two internal hosts
# Host A (server)
iperf3 -s -p 5201

# Host B (client)
iperf3 -c 192.168.1.A -p 5201 -i 1 -t 30

Expected Results by Link Type:

Latency Diagnostics Tools

Ping: Basic Connectivity and Latency Check

ping is the most fundamental network diagnostic tool, sending ICMP Echo Request packets and measuring responses.

Basic Usage:

# Standard ping (Linux/macOS - continuous)
ping 8.8.8.8

# Windows default is 4 packets
ping 8.8.8.8

# Specific number of packets
ping -c 10 8.8.8.8

# Interval and timeout
ping -i 0.2 -W 2 8.8.8.8

# Larger packet size (tests fragmentation)
ping -s 1472 8.8.8.8

Interpreting ping Results:

64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=14.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=14.1 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=14.3 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=117 time=14.2 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=117 time=14.1 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 14.1/14.18/14.3/0.07 ms

Key Metrics:

• time: Round-trip time for each packet
• 0% packet loss: Critical - any loss needs investigation
• rtt mdev: Jitter (standard deviation)
• ttl: Time-to-live remaining (indicates hop count)

Common Issues Detected by ping:

• High latency: time > 100ms suggests network congestion or distance
• Packet loss: loss > 0% indicates network problems
• Variable latency: High mdev suggests jitter problems
• Request timeout: Network path is broken

Traceroute: Path Analysis

traceroute (or tracert on Windows) maps the path packets take to reach a destination, showing each hop.

Basic Usage:

# Linux
traceroute 8.8.8.8

# macOS (may need brew install traceroute)
traceroute 8.8.8.8

# Windows
tracert 8.8.8.8

# Use ICMP (more accurate for modern networks)
traceroute -I 8.8.8.8

# Specify maximum hops
traceroute -m 15 8.8.8.8

# Wait time per probe
traceroute -w 2 8.8.8.8

Sample Output:

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (192.168.1.1)  1.234 ms  1.189 ms  1.167 ms
 2  10.0.0.1 (10.0.0.1)  5.432 ms  5.398 ms  5.376 ms
 3  ISP-core-1.isp.net (203.0.113.1)  10.234 ms  9.876 ms  10.012 ms
 4  IXP-backbone.isp.net (198.51.100.1)  15.678 ms  15.432 ms  15.543 ms
 5  8.8.8.8 (8.8.8.8)  14.234 ms  14.123 ms  14.098 ms

Analyzing Traceroute Results:

• Hop 1: Your local gateway/router
• Consistent increases: Normal - each hop adds latency
• Sudden jumps: Look for the problematic hop
• Asterisks (*): Hop not responding (timeout or firewall blocking ICMP)
• High latency at specific hop: That hop is congested or far away

MTR: Continuous Traceroute with Statistics

MTR combines ping and traceroute, providing continuous path analysis with statistics—essential for identifying intermittent issues.

Installation:

# Ubuntu/Debian
sudo apt install mtr-tiny

# macOS
brew install mtr

# CentOS/RHEL
sudo yum install mtr

Basic Usage:

# Interactive mode (exit with q)
mtr 8.8.8.8

# Report mode (run for 10 cycles then output)
mtr -r -c 10 8.8.8.8

# CSV output for scripting
mtr -c 10 --csv 8.8.8.8

# Wide output (more details)
mtr -r -w 8.8.8.8

Sample Output:

My traceroute  [v0.95]
localhost (0.0.0.0)                 2026-02-24T10:00:00-0000
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                            Packets               Pings
 Host                                                     Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. gateway                                               0.0%   10   1.2   1.3   1.1   1.8  0.2
 2. 10.0.0.1                                              0.0%   10   5.4   5.6   5.2   6.1  0.3
 3. ISP-core-1.isp.net                                    0.0%   10  10.2  10.4   9.8  11.2  0.4
 4. IXP-backbone.isp.net                                   2.0%   10  15.7  16.1  14.9  18.3  1.1
 5. 8.8.8.8                                                0.0%   10  14.2  14.3  14.1  14.6  0.1

Understanding MTR Columns:

• Loss%: Packet loss at each hop (cumulative)
• Snt: Number of packets sent
• Last/Avg/Best/Wrst: Latency statistics
• StDev: Standard deviation (jitter indicator)

MTR Interpretation Guide:

Diagnostic Workflow: Systematic Troubleshooting

Follow this structured approach to diagnose network issues:

Phase 1: Initial Assessment (5 minutes)

Step 1: Confirm the Issue

# Test basic connectivity
ping -c 5 8.8.8.8
ping -c 5 google.com

# If ping to IP works but not DNS
nslookup google.com

Step 2: Gather Baseline Metrics

# Quick speed test
speedtest-cli --simple

# Test to known good server
ping -c 20 1.1.1.1

Questions to Ask:

• Is the issue affecting one user or multiple?
• Is it specific to one application or all network activity?
• When did the issue start?
• What changed recently (updates, new hardware, configuration changes)?

Phase 2: Local Network Diagnosis (10 minutes)

Step 1: Test Local Gateway

# Ping default gateway
ip route | grep default
ping -c 10 <gateway-ip>

# Check local network speed
iperf3 -c <local-server-ip> -p 5201

Step 2: Check Network Interface Stats

# Linux
ip -s link
netstat -i

# Look for errors, drops, overruns
# RX-ERR, RX-DRP, RX-OVR, TX-ERR, TX-DRP, TX-OVR

Step 3: Verify Local Configuration

# Check IP configuration
ip addr show
ip route show

# Check DNS servers
cat /etc/resolv.conf

# Test DNS resolution
dig google.com
time nslookup google.com

Phase 3: ISP and Upstream Diagnosis (15 minutes)

Step 1: Run Continuous Tests

# MTR to external destination
mtr -r -c 50 8.8.8.8

# Look for:
# - Packet loss at any hop
# - High latency at specific hops
# - Jitter (StDev)

Step 2: Speed Tests

# Multiple speed tests to different servers
speedtest-cli --server 1234  # Change server ID
speedtest-cli --server 5678

# Compare results
# Consistent low speeds = ISP throttling or congestion
# Variable speeds = network instability

Step 3: Test Different Protocols

# TCP throughput
iperf3 -c <server> -p 5201

# UDP throughput and packet loss
iperf3 -c <server> -p 5201 -u -b 100M

Phase 4: Application-Specific Diagnosis (varies)

Step 1: Test to Application Endpoint

# For web applications
curl -o /dev/null -s -w "Time: %{time_total}s\n" https://example.com

# For databases
psql -h db.example.com -c "SELECT 1"

# For APIs
curl -X GET https://api.example.com/health

Step 2: Check Application Logs

# Look for timeout errors
grep -i timeout /var/log/app.log

# Connection refused errors
grep -i "connection refused" /var/log/app.log

# DNS resolution failures
grep -i "could not resolve" /var/log/app.log

Differentiating Issue Types

Local Network Issues

Indicators:

• Slow to all external destinations
• Multiple users affected
• Gateway shows high latency or packet loss
• Local network speed tests fail

Common Causes:

• WiFi congestion or interference
• Switch/router problems
• Network cable issues
• VLAN misconfiguration

Commands to Verify:

# Test local gateway
ping -c 10 <gateway-ip>

# Test local server
iperf3 -c 192.168.1.<local-server>

# Check for local network congestion
ip -s link show | grep -E "RX|TX|errors"

ISP Issues

Indicators:

• Speed tests show less than subscribed rate
• High latency to multiple external destinations
• Packet loss visible in MTR
• Issue persists through router (bypassing local network)

Common Causes:

• ISP network congestion
• Local loop issues
• ISP equipment failure
• Peering/transit problems

Commands to Verify:

# Test to multiple external IPs
ping -c 10 8.8.8.8
ping -c 10 1.1.1.1
ping -c 10 208.67.222.222

# Compare results
# Consistent issues across all = ISP problem
# Issue to specific IP only = remote host issue

Application-Level Issues

Indicators:

• Issue specific to one application
• Other applications work fine
• No visible network issues
• Application logs show errors

Common Causes:

• Application configuration
• Firewall rules
• TLS/SSL certificate issues
• DNS resolution specific to application
• Backend service problems

Commands to Verify:

# Test specific endpoint
curl -v https://app.example.com/api

# Test DNS resolution
dig +short app.example.com
nslookup app.example.com

# Check TLS handshake
openssl s_client -connect app.example.com:443

Best Practices for Documentation and Baselines

Establishing Baselines

Create baseline measurements during normal operation to compare against during issues:

Weekly Baseline Script:

#!/bin/bash
# baseline.sh - Network baseline collection

LOGFILE="/var/log/network-baseline.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')

echo "=== Network Baseline $DATE ===" >> "$LOGFILE"

# Latency baseline
echo "Latency tests:" >> "$LOGFILE"
ping -c 20 8.8.8.8 | tail -1 >> "$LOGFILE"
ping -c 20 1.1.1.1 | tail -1 >> "$LOGFILE"

# Speed baseline  
echo "Speed test:" >> "$LOGFILE"
speedtest-cli --simple >> "$LOGFILE"

# Local network baseline
echo "Local network:" >> "$LOGFILE"
iperf3 -c 192.168.1.10 -p 5201 -f M 2>/dev/null | grep "receiver" >> "$LOGFILE"

echo "" >> "$LOGFILE"

Schedule baseline collection:

# Add to crontab
0 8 * * 1 /home/user/scripts/baseline.sh

Incident Documentation Template

Document findings for future reference:

## Network Incident Report

**Date/Time**: YYYY-MM-DD HH:MM
**Reporter**: Name
**Affected Systems**: List

### Symptoms
- Description of what users experienced

### Diagnostics Performed
- ping results: [results]
- traceroute: [results]
- speedtest: [results]
- MTR: [results]

### Root Cause
[Description of what was wrong]

### Resolution
[Steps taken to fix]

### Prevention
[Steps to prevent recurrence]

Continuous Monitoring

Consider deploying continuous monitoring for critical paths:

# Simple monitoring script
#!/bin/bash
# monitor.sh

HOSTS="8.8.8.8 1.1.1.1 208.67.222.222"
THRESHOLD=100  # ms

for host in $HOSTS; do
    result=$(ping -c 5 -W 2 "$host" | tail -1 | awk '{print $4}' | cut -d'/' -f2)
    if (( $(echo "$result > $THRESHOLD" | bc -l) )); then
        echo "ALERT: High latency to $host: $result ms"
    fi
done

Advanced Tips and Techniques

Testing Specific Ports

For application-level diagnostics:

# Test TCP connectivity
nc -zv example.com 443

# Test with timeout
nc -zv -w 5 example.com 443

# Banner grab
nc example.com 80
GET / HTTP/1.0

Analyzing TCP Connection Quality

# Check established connections
ss -tunap

# View TCP statistics
netstat -s | grep -E "Retrans|Outgoing|Failed"

# Extended socket statistics
ss -s

WiFi Diagnostics

# List WiFi networks
nmcli dev wifi list

# Check signal strength
iwconfig

# Check for interference
iwlist wlan0 scanning | grep -E "Channel|Signal|Quality"

Network Namespace for Isolated Testing

For advanced testing in isolated environments:

# Create isolated namespace
ip netns add testns

# Add interface to namespace
ip link set veth0 netns testns

# Run commands in namespace
ip netns exec testns ping 8.8.8.8

Summary

Network troubleshooting requires understanding the four key metrics—bandwidth, latency, packet loss, and jitter—and knowing which tools to use for each. This guide covered:

1. Fundamental Concepts: Understanding bandwidth vs. throughput, latency components, acceptable packet loss thresholds, and jitter impact
2. Bandwidth Testing: Using iperf3 for accurate throughput measurements and speedtest-cli for quick internet speed checks
3. Latency Diagnostics: Employing ping for basic checks, traceroute for path analysis, and MTR for continuous monitoring
4. Systematic Workflow: Following a structured approach from initial assessment through local network, ISP, and application-specific diagnosis
5. Issue Differentiation: Identifying whether problems originate locally, with the ISP, or at the application level
6. Documentation: Establishing baselines and maintaining incident records

Key Takeaways:

• Always baseline your network performance when everything is working
• Use MTR for intermittent issues—it reveals patterns ping misses
• Test multiple destinations to isolate local vs. remote problems
• Document everything—future incidents will be easier to diagnose
• Progress from simple to complex diagnostics

Next Steps:

• Set up baseline monitoring scripts
• Add network diagnostics to your monitoring stack
• Create runbooks for common network issues
• Consider deploying tools like SmokePing for long-term latency tracking

With these techniques and tools, you’re equipped to systematically diagnose and resolve network performance issues, reducing downtime and improving user experience.