Understanding Network Protocols: TCP, UDP, HTTP, DNS, and TLS

Introduction

Network protocols define how applications communicate across the internet. Every HTTP request, database connection, and API call relies on protocol stacks — TCP ensuring reliable delivery, UDP enabling low-latency streaming, DNS translating domain names, and TLS encrypting data in transit. Understanding these protocols helps developers debug connectivity issues, optimize application performance, and build resilient distributed systems.

This guide covers the OSI model, the TCP and UDP transport protocols with packet capture examples, HTTP evolution through HTTP/3, DNS resolution mechanics, TLS handshake, and practical troubleshooting with tcpdump and Wireshark.

OSI Model

The OSI model conceptualizes network communication into seven layers. Each layer provides services to the layer above and consumes services from the layer below. In practice, the TCP/IP model collapses this into four layers (Application, Transport, Internet, Link), but the OSI model remains useful for understanding protocol boundaries.

flowchart LR
    subgraph OSI["OSI Model (7 Layers)"]
        L7["7. Application<br/>HTTP, DNS, SMTP"]
        L6["6. Presentation<br/>TLS, SSL, MIME"]
        L5["5. Session<br/>RPC, NetBIOS"]
        L4["4. Transport<br/>TCP, UDP"]
        L3["3. Network<br/>IP, ICMP, ARP"]
        L2["2. Data Link<br/>Ethernet, WiFi, PPP"]
        L1["1. Physical<br/>Fiber, Coax, Radio"]
    end

    subgraph TCPIP["TCP/IP Model (4 Layers)"]
        A["Application<br/>(L5-L7 combined)"]
        T["Transport<br/>(L4)"]
        I["Internet<br/>(L3)"]
        L["Link<br/>(L1-L2 combined)"]
    end

    OSI --> TCPIP

Transport Layer: TCP vs UDP

The transport layer provides end-to-end communication between applications running on different hosts. TCP and UDP serve fundamentally different purposes.

TCP Three-Way Handshake

Every TCP connection begins with a three-way handshake that synchronizes sequence numbers and establishes window sizes:

sequenceDiagram
    participant Client
    participant Server

    Client->>Server: SYN (seq=x)
    Note over Client: SYN_SENT state
    Server-->>Client: SYN-ACK (seq=y, ack=x+1)
    Note over Server: SYN_RCVD state
    Client->>Server: ACK (seq=x+1, ack=y+1)
    Note over Client,Server: ESTABLISHED state
    Note over Client,Server: Data transfer begins

TCP Connection Termination

Connections close with a four-way FIN handshake:

sequenceDiagram
    participant Client
    participant Server

    Client->>Server: FIN (seq=x)
    Server-->>Client: ACK (seq=y, ack=x+1)
    Server-->>Client: FIN (seq=z, ack=x+1)
    Client->>Server: ACK (seq=x+1, ack=z+1)
    Note over Client: TIME_WAIT state (2MSL)

Capturing TCP Handshake with tcpdump

# Capture a TCP handshake to google.com on interface eth0
sudo tcpdump -i eth0 -nn 'host 142.250.80.46 and tcp port 443'

# Output (synthetic):
# 14:23:01.123456 IP 10.0.0.5.54321 > 142.250.80.46.443: Flags [S], seq 1234567890
# 14:23:01.234567 IP 142.250.80.46.443 > 10.0.0.5.54321: Flags [S.], seq 9876543210, ack 1234567891
# 14:23:01.234678 IP 10.0.0.5.54321 > 142.250.80.46.443: Flags [.], ack 9876543211

# Flags legend: [S] = SYN, [S.] = SYN-ACK, [.] = ACK, [F] = FIN, [R] = RST

When to Use TCP vs UDP

Use TCP when reliability matters more than latency: HTTP APIs, database connections (PostgreSQL, MySQL), message queues (Kafka, RabbitMQ), email delivery (SMTP, IMAP). TCP’s retransmission and ordering guarantees ensure data integrity at the cost of head-of-line blocking.

Use UDP when latency matters more than reliability: live video streaming (WebRTC), DNS queries, online gaming, VoIP calls, QUIC/HTTP/3. UDP’s zero-overhead datagram model avoids head-of-line blocking but requires application-level retry logic when needed.

HTTP Evolution

HTTP/1.1 (1997)

• Persistent connections (keep-alive)
• Pipelining (limited adoption)
• Text-based protocol
• Head-of-line blocking per connection

HTTP/2 (2015)

# h2c (HTTP/2 cleartext) upgrade
# Client sends:
GET / HTTP/1.1
Host: example.com
Upgrade: h2c
HTTP2-Settings: <base64>

# Server responds with 101 Switching Protocols
# Then communicates in HTTP/2 binary frames

• Binary framing layer (streams, frames, messages)
• Multiplexing multiple requests over one connection
• Server push
• Header compression (HPACK)
• Single TCP connection eliminates head-of-line blocking at the application layer, but TCP-level HOL blocking still occurs

HTTP/3 and QUIC (2022+)

HTTP/3 replaces TCP with QUIC, which runs over UDP. QUIC was standardized as RFC 9000 in 2021, and HTTP/3 (RFC 9114) followed in 2022. By 2026, HTTP/3 accounts for over 40% of web traffic.

sequenceDiagram
    participant Client
    participant Server

    Note over Client,Server: QUIC handshake (0-RTT if cached)
    Client->>Server: Initial (ClientHello, QUIC version, source connection ID)
    Server-->>Client: Handshake (ServerHello, certificate, transport params)
    Note over Client,Server: TLS 1.3 handshake completes within QUIC

    Client->>Server: 1-RTT (HTTP/3 HEADERS frame: GET /)
    Server-->>Client: 1-RTT (HTTP/3 DATA frame: response body)
    Note over Client,Server: QUIC handles loss per-stream, no HOL blocking

    Client->>Server: 0-RTT (cached session resumption, no handshake!)
    Server-->>Client: 1-RTT (immediate data response)

Key advantages over HTTP/2:

• No head-of-line blocking: A lost packet affects only its stream, not all streams on the connection
• 0-RTT handshake: Repeat connections can send data immediately using cached session state
• Connection migration: Connection survives IP address changes (mobile transitioning from WiFi to cellular)

TLS Handshake

TLS 1.3 (RFC 8446, 2018) reduced the handshake from two round trips to one, improving latency significantly:

sequenceDiagram
    participant Client
    participant Server

    Client->>Server: ClientHello (key_share, signature_algorithms, supported versions)
    Server-->>Client: ServerHello + EncryptedExtensions + Certificate + CertificateVerify + Finished
    Note over Client,Server: 1-RTT complete

    Client->>Server: Finished + Application Data (HTTP request)
    Note over Client,Server: Encrypted communication established in 1 round trip

TLS 1.3 with tcpdump and Wireshark

# Check TLS version and cipher for a connection
sudo tcpdump -i eth0 -nn 'tcp port 443' -c 20 -X

# Use ss to see TLS parameters on established connections
# (requires kernel TLS support)
ss -tnp | head -20

# For detailed TLS analysis, capture and open in Wireshark
sudo tcpdump -i eth0 -nn -w tls.cap 'tcp port 443'
# Then open tls.cap in Wireshark and follow TLS stream

Common Wireshark Display Filters for Protocol Analysis

# Show only TCP handshake packets (SYN, SYN-ACK, ACK)
tcp.flags.syn == 1 and tcp.flags.ack == 0

# Show TCP retransmissions (indicates packet loss)
tcp.analysis.retransmission

# Show TLS 1.3 handshakes
tls.handshake.type == 1

# Filter HTTP/3 (QUIC) traffic
quic

# Show DNS queries and responses
dns

# Show packets with RST (connection reset)
tcp.flags.reset == 1

DNS Resolution

DNS translates human-readable domain names into IP addresses. The resolution process involves multiple layers of caching and delegation:

sequenceDiagram
    participant Browser
    participant Cache as Local DNS Cache
    participant Resolver as DNS Resolver<br/>(ISP or 8.8.8.8)
    participant Root as Root DNS Server
    participant TLD as .com TLD Server
    participant Auth as Authoritative Server<br/>(example.com)

    Browser->>Cache: query "example.com"
    Cache-->>Browser: miss (not cached)
    Browser->>Resolver: query "example.com"
    Resolver->>Root: where is .com?
    Root-->>Resolver: refer to a.gtld-servers.net
    Resolver->>TLD: where is example.com?
    TLD-->>Resolver: ns1.example.com (authoritative)
    Resolver->>Auth: what is A record for example.com?
    Auth-->>Resolver: A record: 93.184.216.34
    Resolver->>Cache: cache result (TTL=300)
    Resolver-->>Browser: 93.184.216.34
    Browser->>Auth: HTTP GET /

DNS Record Types

DNS Troubleshooting with dig

# Basic A record lookup
dig example.com

# Short answer only
dig +short example.com

# Query a specific resolver (Google's 8.8.8.8)
dig @8.8.8.8 example.com

# Trace the full resolution path
dig +trace example.com

# Query for MX records
dig mx example.com

# Check CNAME resolution
dig cname www.example.com

# Reverse DNS lookup (IP to name)
dig -x 93.184.216.34

# Measure query time
dig example.com | grep "Query time"

Security Considerations

Common Network Attacks

TCP SYN Flood: Attacker sends many SYN packets without completing the handshake, exhausting server connection table. Mitigation: SYN cookies (enabled by default in Linux).

DNS Spoofing / Cache Poisoning: Attacker injects fake DNS responses into a resolver’s cache. Mitigation: DNSSEC (DNS Security Extensions), DNS-over-TLS, DNS-over-HTTPS.

Man-in-the-Middle: Attacker intercepts and modifies traffic between client and server. Mitigation: TLS with certificate pinning, HSTS.

Network Security Commands

# Check if SYN cookies are enabled
sysctl net.ipv4.tcp_syncookies

# Verify DNSSEC for a domain
dig example.com +dnssec

# Check which DNS resolver you're using
resolvectl status

# List all listening ports and their services
sudo ss -tlnp

# Scan for open ports on a remote host (basic connectivity check)
nc -zv example.com 443
nc -zv example.com 80

# Test UDP connectivity (DNS query via nc)
echo "example.com" | nc -u -w2 8.8.8.8 53

Troubleshooting Tool Reference

Resources

• RFC 9293 — Transmission Control Protocol (TCP)
• RFC 9000 — QUIC: A UDP-Based Multiplexed and Secure Transport
• RFC 9114 — HTTP/3
• RFC 8446 — TLS 1.3
• tcpdump Man Page — Packet capture reference
• Wireshark Display Filter Reference — Protocol filter reference
• Cloudflare Learning Center — Network protocol explanations