Go’s concurrency primitives are fast by design — goroutines are cheap, channels are efficient, and the scheduler is work-stealing. But “cheap” doesn’t mean “free,” and it’s easy to write concurrent code that’s slower than its single-threaded equivalent. The culprits are almost always: too many goroutines, lock contention, false sharing, or too many allocations in the hot path.
This guide covers the techniques for finding and fixing concurrency bottlenecks. For foundational patterns see Go worker pools, Go sync package, and Go profiling.
Profiling First: Find the Real Bottleneck
Before tuning anything, measure. The Go profiler (pprof) shows you exactly where time and memory are going. Enable it in any HTTP service by importing the side-effect package:
import _ "net/http/pprof" // registers /debug/pprof/ handlers automatically
go http.ListenAndServe(":6060", nil)
Then profile a running service:
# CPU profile — what code is consuming CPU time
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
# Goroutine dump — what every goroutine is doing right now
go tool pprof http://localhost:6060/debug/pprof/goroutine
# Mutex contention profile — which mutexes are the hottest
go tool pprof http://localhost:6060/debug/pprof/mutex
# Block profile — goroutines blocked on channel or mutex operations
go tool pprof http://localhost:6060/debug/pprof/block
The mutex and block profiles require opt-in at runtime:
runtime.SetMutexProfileFraction(1) // profile 100% of mutex contention events
runtime.SetBlockProfileRate(1) // profile every blocking operation
In production, use a fraction (e.g., SetMutexProfileFraction(10) samples 1 in 10 events) to limit overhead. The go tool pprof interactive shell and pprof -http=:8080 web UI both show flamegraphs that make hotspots obvious.
Goroutine Count: More Is Not Better
The most common concurrency performance mistake is spawning too many goroutines. Each goroutine has a minimum 2–8 KB stack. 100,000 goroutines × 4 KB = 400 MB of stack alone, before any heap allocations. More goroutines also means more work for the scheduler, and more context switches.
CPU-bound work: cap goroutines at runtime.NumCPU(). Beyond that, you’re adding scheduling overhead without adding parallelism — the CPU is already at full utilization.
I/O-bound work: goroutines spend most of their time blocked on network or disk, so you can run many more. But the right number depends on the downstream system — a database can usually handle 20–50 concurrent queries before latency degrades significantly. Start conservative and benchmark.
The worker pool pattern enforces this cap:
numWorkers := runtime.NumCPU() // for CPU-bound
// numWorkers := 20 // for I/O-bound — tune per downstream
jobs := make(chan Job, numWorkers*2)
var wg sync.WaitGroup
for i := 0; i < numWorkers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for job := range jobs {
process(job)
}
}()
}
Check goroutine count in tests with runtime.NumGoroutine() and use goleak to catch leaks — goroutines that block forever are a memory leak that grows over time.
Channel Buffer Sizing
An unbuffered channel synchronizes sender and receiver — each send blocks until a receiver is ready. This is correct for some patterns (handshakes, notifications) but is a throughput bottleneck when you want one producer feeding multiple workers.
A buffered channel decouples producer and consumer: the producer can send up to the buffer capacity without waiting. The right buffer size depends on the pattern:
- Job queue feeding a worker pool: buffer of
numWorkersornumWorkers * 2. This keeps all workers busy without queuing too much backlog. - Result collection channel: buffer of the maximum expected result count, so workers never block writing results.
- Event notification: buffer of 1 is often sufficient — the receiver will eventually drain it.
// Jobs: buffer keeps workers fed during bursts
jobs := make(chan Job, numWorkers*2)
// Results: pre-sized so workers never block
results := make(chan Result, totalJobCount)
The cost of too-large a buffer is memory; the cost of too-small is blocked goroutines. For a job queue, erring toward larger is safer since job structs are usually small.
Atomic Operations: Faster Than Mutexes for Simple State
For single-variable state that only needs increment, load, or compare-and-swap, the sync/atomic package is significantly faster than a mutex. Atomics use CPU-level instructions that don’t involve the Go scheduler at all:
import "sync/atomic"
type Stats struct {
requests atomic.Int64
errors atomic.Int64
latencyNs atomic.Int64
}
func (s *Stats) RecordRequest(latency time.Duration, err error) {
s.requests.Add(1)
s.latencyNs.Add(latency.Nanoseconds())
if err != nil {
s.errors.Add(1)
}
}
func (s *Stats) Report() {
reqs := s.requests.Load()
fmt.Printf("requests: %d, errors: %d\n", reqs, s.errors.Load())
}
Go 1.19 introduced the typed atomic types (atomic.Int64, atomic.Bool, atomic.Pointer[T]) which are cleaner than the older function-based API (atomic.AddInt64(&v, 1)). Use the typed API for new code.
When atomics aren’t enough: if you need to atomically update multiple related values (e.g., both a count and a total for computing a mean), you need a mutex — there’s no way to atomically update two separate memory locations. Atomics only work for single-variable operations.
Reducing Lock Contention
Lock contention happens when many goroutines compete for the same mutex, forcing most of them to wait. The fix is either to hold the lock for less time, or to reduce the number of goroutines competing.
Move work outside the critical section. Compute what you need before acquiring the lock:
// ❌ Lock held during expensive computation
func (c *Cache) GetOrCompute(key string) string {
c.mu.Lock()
defer c.mu.Unlock()
if v, ok := c.data[key]; ok {
return v
}
v := expensiveCompute(key) // slow — lock held the whole time
c.data[key] = v
return v
}
// ✅ Check under read lock, compute outside, write under write lock
func (c *Cache) GetOrCompute(key string) string {
c.mu.RLock()
if v, ok := c.data[key]; ok {
c.mu.RUnlock()
return v
}
c.mu.RUnlock()
v := expensiveCompute(key) // no lock held here
c.mu.Lock()
if existing, ok := c.data[key]; ok { // re-check after compute
c.mu.Unlock()
return existing // another goroutine computed it first
}
c.data[key] = v
c.mu.Unlock()
return v
}
The double-check after expensiveCompute handles the race between two goroutines that both found a cache miss — only one should win the write.
Shard your data structures. Instead of one mutex for an entire map, use N mutexes for N shards. Each key maps to a shard via hash, so contention is divided by N:
const numShards = 64
type ShardedMap struct {
shards [numShards]struct {
mu sync.RWMutex
data map[string]string
}
}
func (m *ShardedMap) shard(key string) int {
h := fnv.New32a()
h.Write([]byte(key))
return int(h.Sum32()) % numShards
}
func (m *ShardedMap) Set(key, value string) {
s := m.shard(key)
m.shards[s].mu.Lock()
m.shards[s].data[key] = value
m.shards[s].mu.Unlock()
}
func (m *ShardedMap) Get(key string) (string, bool) {
s := m.shard(key)
m.shards[s].mu.RLock()
v, ok := m.shards[s].data[key]
m.shards[s].mu.RUnlock()
return v, ok
}
With 64 shards, on average 64 goroutines can read or write simultaneously without contention. Libraries like github.com/orcaman/concurrent-map implement this pattern with more features.
Batch Processing for Throughput
Processing items one at a time adds per-item overhead: channel operations, mutex acquisitions, function calls. Batching amortizes this cost across many items:
// Individual writes: one channel op and one DB call per item
for _, item := range items {
writeCh <- item
}
// Batched writes: one channel op and one DB call per batch
func batcher(in <-chan Item, batchSize int, flush func([]Item)) {
batch := make([]Item, 0, batchSize)
ticker := time.NewTicker(100 * time.Millisecond) // flush timeout
defer ticker.Stop()
for {
select {
case item, ok := <-in:
if !ok {
if len(batch) > 0 {
flush(batch)
}
return
}
batch = append(batch, item)
if len(batch) >= batchSize {
flush(batch)
batch = batch[:0]
}
case <-ticker.C:
if len(batch) > 0 {
flush(batch)
batch = batch[:0]
}
}
}
}
The ticker ensures a partial batch is flushed within the timeout even if the batch size threshold is never reached. This is the pattern used by log aggregators, metrics collectors, and write-ahead log buffers in databases.
False Sharing: An Easy Miss
False sharing is a CPU cache phenomenon where two goroutines on different cores modify different variables that happen to sit in the same 64-byte cache line. The CPU treats them as conflicting writes even though they’re logically independent, causing cache invalidation traffic between cores.
The fix is padding:
// ❌ False sharing: counter[0] and counter[1] likely share a cache line
type Counters struct {
values [4]int64
}
// ✅ Each counter is in its own cache line
type PaddedCounter struct {
value int64
_ [56]byte // pad to 64 bytes (cache line size)
}
type Counters struct {
c [4]PaddedCounter
}
False sharing typically shows up in benchmarks as unexpected scaling degradation — adding CPUs makes things slower. The perf c2c Linux tool can detect it. In practice it’s rare in Go code unless you’re writing per-CPU accumulators or similar structures.
Benchmark to Validate Changes
Every optimization claim should be backed by a benchmark. Go’s testing.B makes this straightforward:
func BenchmarkMutex(b *testing.B) {
var mu sync.Mutex
v := 0
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
mu.Lock()
v++
mu.Unlock()
}
})
}
func BenchmarkAtomic(b *testing.B) {
var v atomic.Int64
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
v.Add(1)
}
})
}
go test -bench=. -benchtime=5s -count=3 | tee results.txt
benchstat results.txt # from golang.org/x/perf/cmd/benchstat
b.RunParallel runs the inner function from multiple goroutines simultaneously, which is essential for testing concurrent code — single-goroutine benchmarks don’t show contention.
Summary
- Profile before optimizing —
pprof’s mutex and block profiles show exactly where contention is - Cap goroutine count:
runtime.NumCPU()for CPU-bound, 10–50 for I/O-bound - Size channel buffers to match producer/consumer patterns —
numWorkers*2for job queues - Use
sync/atomictyped values for single-variable counters and flags — much faster than a mutex - Move computation outside critical sections; consider sharding hot maps across 16–64 mutexes
- Batch small writes into larger units to amortize per-operation overhead
- Always benchmark changes with
b.RunParallel— optimizations that look good on paper can regress under real contention
Resources
- Go pprof documentation
- sync/atomic package
- Go Memory Model
- Profiling Go Programs (Go Blog)
- benchstat
Comments