Bytes, Runes, and Unicode in Go

Go strings are sequences of bytes, not characters. This is the single most important thing to understand about text in Go, and it’s the source of most Unicode-related bugs. A string containing the Chinese character 世 is 3 bytes long but 1 character long. len("世") returns 3, not 1. Indexing s[0] gives you the first byte, not the first character.

This design is intentional — it keeps strings simple and efficient for the common case (ASCII), while the unicode/utf8 package and rune type provide the tools to handle multi-byte characters correctly when needed.

For string manipulation see Go strings package. For text processing algorithms see Go text processing.

The Model: Strings, Bytes, and Runes

A string in Go is an immutable sequence of bytes. It has no encoding attached — the convention is UTF-8, but the compiler doesn’t enforce it.

A byte (uint8) is a single 8-bit value. ASCII characters are single bytes; most other Unicode characters require 2–4 bytes in UTF-8.

A rune (int32) represents a Unicode code point — a single character regardless of how many bytes it takes to encode. The word “rune” is Go’s name for what Unicode calls a “code point.”

s := "Hello, 世界"

fmt.Println(len(s))          // 13 — bytes (7 ASCII + 3 + 3 for the two Chinese chars)
fmt.Println(len([]rune(s)))  // 9 — characters (7 ASCII + 2 Chinese)

// Indexing a string gives a byte, not a character
fmt.Printf("%T %v\n", s[0], s[0])  // uint8 72 ('H')

// The two Chinese characters start at byte index 7
fmt.Printf("%T %v\n", s[7], s[7])  // uint8 228 (first byte of '世', NOT '世' itself)

Iterating: Bytes vs Runes

Iterating with an index (for i := 0; i < len(s); i++) walks bytes — each iteration gives s[i], a uint8. This is correct for ASCII-only strings but breaks on any multi-byte character:

s := "café"
for i := 0; i < len(s); i++ {
    fmt.Printf("s[%d] = %d\n", i, s[i])
}
// s[0] = 99  (c)
// s[1] = 97  (a)
// s[2] = 102 (f)
// s[3] = 195 (first byte of é — NOT 'é')
// s[4] = 169 (second byte of é)

Iterating with range decodes UTF-8 automatically and gives you each character as a rune along with its byte index:

s := "café"
for i, r := range s {
    fmt.Printf("s[%d] = %c (%U)\n", i, r, r)
}
// s[0] = c (U+0063)
// s[1] = a (U+0061)
// s[2] = f (U+0066)
// s[3] = é (U+00E9)  ← byte index jumps from 3 to 5 (é is 2 bytes)

The byte index i is where the character starts in the string. Notice the jump from index 3 to 5 — é occupies bytes 3 and 4, so the next character would start at byte index 5.

Use range for character-level iteration. Use index iteration only when you know the string is ASCII or when you explicitly need byte-level access.

The unicode/utf8 Package

The unicode/utf8 package exposes the UTF-8 encoding details directly:

import "unicode/utf8"

s := "Hello, 世界"

// Character count (not byte count)
fmt.Println(utf8.RuneCountInString(s))  // 9

// Decode the first rune and its byte width
r, size := utf8.DecodeRuneInString(s[7:])
fmt.Printf("rune: %c, bytes: %d\n", r, size)  // rune: 世, bytes: 3

// Validate that a string is well-formed UTF-8
fmt.Println(utf8.ValidString("valid"))           // true
fmt.Println(utf8.Valid([]byte{0xFF, 0xFE}))      // false — invalid UTF-8 bytes

// Encode a rune to its UTF-8 bytes
buf := make([]byte, utf8.UTFMax)
n := utf8.EncodeRune(buf, '世')
fmt.Println(buf[:n], n)  // [228 184 150] 3

utf8.RuneCountInString is the correct way to measure “how many characters” — not len. utf8.ValidString belongs in any function that accepts string input from outside the process (user input, files, network).

Converting Between String, []byte, and []rune

Each conversion has a cost — []byte(s) and []rune(s) both allocate and copy. In hot paths, minimize conversions:

s := "Hello, 世界"

// String → []byte: copy, O(n)
// Use for: I/O, bytes.Buffer, any []byte API
b := []byte(s)

// String → []rune: decode + copy, O(n), allocates n rune-sized slots
// Use for: character-level indexing/slicing/reversal
r := []rune(s)

// []byte → string: copy, O(n)
s2 := string(b)

// []rune → string: encode + copy, O(n)
s3 := string(r)

If you only need to iterate character by character, range is faster than converting to []rune first — it decodes on the fly without an allocation.

If you need character-indexed slicing (e.g., “give me characters 3–7”), convert to []rune:

s := "Hello, 世界"
runes := []rune(s)
fmt.Println(string(runes[7:9]))  // 世界

You can’t do s[7:9] because the byte indices for characters aren’t the same as character indices in non-ASCII strings.

The unicode Package: Character Classification

The unicode package provides predicates for character categories, following the Unicode standard:

import "unicode"

// Character classification
unicode.IsLetter('A')    // true — any Unicode letter, not just ASCII
unicode.IsDigit('5')     // true
unicode.IsSpace(' ')     // true (also \t, \n, \r, and Unicode space chars)
unicode.IsUpper('A')     // true
unicode.IsLower('a')     // true
unicode.IsPunct(',')     // true

// Case conversion
unicode.ToUpper('é')     // 'É' — handles Unicode, not just ASCII
unicode.ToLower('Ç')     // 'ç'

// Range tables for script-specific checks
unicode.Is(unicode.Latin, 'A')   // true
unicode.Is(unicode.Han, '世')   // true — Chinese/Japanese/Korean unified ideographs

unicode.IsLetter returns true for letters from any script — Arabic, Cyrillic, Chinese, etc. This is the correct check for “is this a word character” in a Unicode-aware application, as opposed to r >= 'A' && r <= 'Z' || r >= 'a' && r <= 'z' which only covers ASCII Latin.

Practical: Reversing a Unicode String

A classic example that shows why bytes vs runes matters:

// ❌ Reverses bytes — mangles multi-byte characters
func reverseBytes(s string) string {
    b := []byte(s)
    for i, j := 0, len(b)-1; i < j; i, j = i+1, j-1 {
        b[i], b[j] = b[j], b[i]
    }
    return string(b)
}

// ✅ Reverses characters correctly
func reverseString(s string) string {
    runes := []rune(s)
    for i, j := 0, len(runes)-1; i < j; i, j = i+1, j-1 {
        runes[i], runes[j] = runes[j], runes[i]
    }
    return string(runes)
}

fmt.Println(reverseBytes("Hello, 世界"))   // garbled output
fmt.Println(reverseString("Hello, 世界")) // 界世 ,olleH

Practical: Counting Specific Character Categories

func classify(s string) (letters, digits, spaces, other int) {
    for _, r := range s {  // range gives runes
        switch {
        case unicode.IsLetter(r):
            letters++
        case unicode.IsDigit(r):
            digits++
        case unicode.IsSpace(r):
            spaces++
        default:
            other++
        }
    }
    return
}

l, d, sp, o := classify("Hello, 世界 42!")
fmt.Printf("letters=%d digits=%d spaces=%d other=%d\n", l, d, sp, o)
// letters=7 digits=2 spaces=1 other=2

Practical: Validating External Input

Any string arriving from outside the process — HTTP request body, file read, database query — might contain invalid UTF-8 bytes. Validate before processing if your code assumes valid UTF-8:

func processUserInput(input string) error {
    if !utf8.ValidString(input) {
        return fmt.Errorf("input contains invalid UTF-8")
    }
    // safe to use range iteration and rune operations
    for _, r := range input {
        if r == utf8.RuneError {
            // RuneError (U+FFFD) can appear in valid strings as the replacement character
            // or when decoding bad input — handle appropriately
        }
        _ = r
    }
    return nil
}

utf8.RuneError (U+FFFD, the Unicode replacement character) is what range yields when it encounters bytes that don’t form a valid UTF-8 sequence. Checking for it lets you handle encoding errors explicitly.

Quick Reference

Summary

• len(s) counts bytes, not characters — use utf8.RuneCountInString(s) for character count
• range on a string iterates runes (characters) with their byte-start position — use it for character-level processing
• Convert to []rune only when you need character-indexed access (slicing, reversal) — range is cheaper for iteration
• unicode.IsLetter, unicode.IsDigit, etc. handle all Unicode scripts, not just ASCII
• Validate external string input with utf8.ValidString before assuming it’s well-formed

Resources

• Go Blog: Strings, bytes, runes and characters in Go
• unicode/utf8 package
• unicode package
• UTF-8 specification