Service Discovery and Load Balancing in Go

Introduction

Service discovery and load balancing are critical components of microservices architecture. As services scale horizontally, you need mechanisms to discover available service instances and distribute traffic efficiently. This guide covers implementing these patterns in Go. See Go Installation Guide, Go Ecosystem Overview, Go Best Practices for more context.

Service discovery allows services to find and communicate with each other dynamically, while load balancing distributes requests across multiple instances to ensure optimal resource utilization and high availability. Both problems appear together in practice: you cannot balance traffic across instances you have not discovered, and there is little point discovering instances if you have no strategy for choosing among them. The examples in this article build on each other, so read them in order.

Service Discovery Patterns

Discovery mechanisms fall into two broad families: those that reuse existing infrastructure, such as DNS, and those that use a dedicated registry, such as Consul or etcd. The choice between them is a classic trade-off between operational simplicity and feature richness. DNS requires no new components but is slow to propagate changes and carries no health information. A registry is a new stateful component to operate, but it gives you health checks, metadata, and near-real-time updates. Go’s standard library makes both approaches easy to implement, as the next two sections show.

DNS-Based Discovery

DNS-based discovery is the oldest and simplest approach. It leans on infrastructure that almost certainly already exists in your environment. The Go code below uses the standard net.Resolver to look up a service name and return the set of addresses currently registered for it. Two methods are shown, and each solves a different problem. DiscoverService calls LookupHost, which returns IP addresses but no port information. DiscoverServiceWithSRV calls LookupSRV, which returns both the target host and the service port — the crucial detail when a service name maps to multiple instances on different ports.

The code illustrates two design decisions that carry across every discovery implementation in this article. First, both lookups wrap the context with a five-second timeout. A DNS server that hangs must not hang your request path, so the timeout bounds the failure latency. Second, the SRV method composes the host and port into a single host:port string, because that is the format the rest of the Go networking stack and HTTP clients expect. The main weakness of DNS discovery is not visible in the code: cached DNS answers can serve stale addresses for a long time after an instance dies, and there is no way to ask DNS whether an address is currently healthy.

package main

import (
	"context"
	"fmt"
	"net"
	"time"
)

// DNSDiscovery uses DNS for service discovery
type DNSDiscovery struct {
	resolver *net.Resolver
}

// NewDNSDiscovery creates a new DNS discovery
func NewDNSDiscovery() *DNSDiscovery {
	return &DNSDiscovery{
		resolver: net.DefaultResolver,
	}
}

// DiscoverService discovers service instances via DNS
func (d *DNSDiscovery) DiscoverService(ctx context.Context, serviceName string) ([]string, error) {
	// Set timeout for DNS lookup
	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
	defer cancel()

	// Perform DNS lookup
	addrs, err := d.resolver.LookupHost(ctx, serviceName)
	if err != nil {
		return nil, fmt.Errorf("DNS lookup failed: %w", err)
	}

	return addrs, nil
}

// DiscoverServiceWithSRV discovers service instances via SRV records
func (d *DNSDiscovery) DiscoverServiceWithSRV(ctx context.Context, service, proto, name string) ([]string, error) {
	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
	defer cancel()

	_, srvs, err := d.resolver.LookupSRV(ctx, service, proto, name)
	if err != nil {
		return nil, fmt.Errorf("SRV lookup failed: %w", err)
	}

	var addresses []string
	for _, srv := range srvs {
		address := fmt.Sprintf("%s:%d", srv.Target, srv.Port)
		addresses = append(addresses, address)
	}

	return addresses, nil
}

DNS discovery works best for small or medium deployments where a little staleness is acceptable. Kubernetes, for instance, uses a DNS-based discovery model internally for exactly this reason. Its simplicity is also its appeal: there is no separate registry to operate, no agent to install, and no consistency protocol to debug. Use it when your service name to instance mapping changes slowly and you can tolerate minutes of propagation delay. Reach for a dedicated registry when you need health-aware selection or sub-second failover.

Consul-Based Discovery

When DNS staleness becomes a problem, teams typically move to a service registry such as Consul, etcd, or ZooKeeper. The ConsulClient below sketches the shape of a registry-backed client. Its ServiceInstance struct carries the fields a registry returns — an ID, name, address, port, tags, and arbitrary metadata — which is already more information than DNS can provide. The registration methods show the lifecycle contract: services register when they start, deregister when they shut down cleanly, and rely on periodic health checks to prune instances that fail without deregistering. The DiscoverService method returns the current healthy instances for a name.

The most interesting design decision is WatchService, which turns a one-shot query into a stream of updates. It spawns a goroutine that polls DiscoverService on a five-second ticker and pushes results onto a channel that consumers range over. The context cancels the goroutine cleanly, so the caller controls the lifetime. This channel-based watch is the idiomatic Go way to expose change notifications, and it is the mechanism that lets a load balancer react to instances joining or leaving the cluster. The polling interval is a tuning knob: faster polling means fresher views but more load on the registry.

package main

import (
	"context"
	"fmt"
	"time"
)

// ConsulClient represents a Consul client
type ConsulClient struct {
	baseURL string
	client  interface{} // Would be actual Consul client
}

// ServiceInstance represents a service instance in Consul
type ServiceInstance struct {
	ID      string
	Name    string
	Address string
	Port    int
	Tags    []string
	Meta    map[string]string
}

// RegisterService registers a service with Consul
func (c *ConsulClient) RegisterService(ctx context.Context, instance ServiceInstance) error {
	// Implementation would call Consul API
	fmt.Printf("Registering service: %s at %s:%d\n", instance.Name, instance.Address, instance.Port)
	return nil
}

// DeregisterService deregisters a service from Consul
func (c *ConsulClient) DeregisterService(ctx context.Context, serviceID string) error {
	fmt.Printf("Deregistering service: %s\n", serviceID)
	return nil
}

// DiscoverService discovers service instances from Consul
func (c *ConsulClient) DiscoverService(ctx context.Context, serviceName string) ([]ServiceInstance, error) {
	// Implementation would call Consul API
	instances := []ServiceInstance{
		{
			ID:      "service-1",
			Name:    serviceName,
			Address: "192.168.1.1",
			Port:    8080,
		},
		{
			ID:      "service-2",
			Name:    serviceName,
			Address: "192.168.1.2",
			Port:    8080,
		},
	}
	return instances, nil
}

// WatchService watches for service changes
func (c *ConsulClient) WatchService(ctx context.Context, serviceName string) (<-chan []ServiceInstance, error) {
	ch := make(chan []ServiceInstance)

	go func() {
		defer close(ch)
		ticker := time.NewTicker(5 * time.Second)
		defer ticker.Stop()

		for {
			select {
			case <-ctx.Done():
				return
			case <-ticker.C:
				instances, err := c.DiscoverService(ctx, serviceName)
				if err == nil {
					ch <- instances
				}
			}
		}
	}()

	return ch, nil
}

The watch pattern is worth abstracting into a reusable utility. Any client that consumes a <-chan []ServiceInstance can ignore the mechanics of polling, backoff, and cancellation. Keep that channel unidirectional so consumers cannot send into it. The trade-off to remember is that registry-based discovery trades DNS simplicity for operational responsibility. A registry is another stateful service that needs backup, monitoring, and capacity planning, so only adopt it when the freshness and health guarantees justify the cost.

Load Balancing Strategies

With discovery solved, the next problem is choosing which instance receives each request. Load balancing strategies differ in how much state they track and how evenly they distribute traffic. The three strategies below represent the standard progression. Round-robin requires no per-instance state and gives a perfectly even distribution in the average case. Least-connections tracks live connection counts and routes around busy instances. Weighted balancing injects explicit capacity ratios, so you can send more traffic to a larger instance. Each one trades a small amount of bookkeeping for better adaptivity.

Round-Robin Load Balancer

Round-robin is the baseline strategy: walk through the instance list in order, wrapping around when you reach the end. The RoundRobinBalancer below is the canonical Go implementation, and its structure highlights two concurrency concerns that apply to every balancer in this article. First, the counter is incremented with atomic.AddUint64, so concurrent callers never observe the same index. Second, the instance slice is guarded by a read-write mutex because UpdateInstances can replace it while other goroutines are selecting from it. The RLock in SelectInstance allows many concurrent reads while ensuring a writer waits for all of them to finish.

The selection math is deliberately simple: counter % len(instances) picks the next slot. That modulo is what produces the even distribution, and it automatically adapts when UpdateInstances changes the slice length. The error path matters too — selecting from an empty instance list returns a descriptive error rather than panicking on a modulo by zero. Round-robin’s weakness is that it treats every instance as equally capable. If one instance is slower or under heavier CPU pressure, round-robin keeps sending it a fair share of requests anyway, which is exactly the problem the next strategy addresses.

package main

import (
	"context"
	"fmt"
	"sync"
	"sync/atomic"
)

// RoundRobinBalancer implements round-robin load balancing
type RoundRobinBalancer struct {
	instances []string
	counter   uint64
	mu        sync.RWMutex
}

// NewRoundRobinBalancer creates a new round-robin balancer
func NewRoundRobinBalancer(instances []string) *RoundRobinBalancer {
	return &RoundRobinBalancer{
		instances: instances,
		counter:   0,
	}
}

// SelectInstance selects the next instance
func (b *RoundRobinBalancer) SelectInstance(ctx context.Context) (string, error) {
	b.mu.RLock()
	defer b.mu.RUnlock()

	if len(b.instances) == 0 {
		return "", fmt.Errorf("no instances available")
	}

	index := atomic.AddUint64(&b.counter, 1) - 1
	return b.instances[index%uint64(len(b.instances))], nil
}

// UpdateInstances updates the list of instances
func (b *RoundRobinBalancer) UpdateInstances(instances []string) {
	b.mu.Lock()
	defer b.mu.Unlock()
	b.instances = instances
}

Round-robin is the right default for most systems. It is stateless, trivially fast, and perfectly fair in aggregate. Use it when instances are homogeneous or when you have no visibility into each instance’s current load. The moment instances become heterogeneous in capacity or utilization, upgrade to a strategy that observes load, starting with least-connections.

Least Connections Load Balancer

Least-connections balancing sends each request to the instance currently handling the fewest active connections. The LeastConnectionsBalancer below tracks a connection count per address in an InstanceMetrics struct. Selection is a linear scan over the map that keeps the instance with the smallest Connections value. The logic is O(n) rather than O(1), but for typical fleet sizes of tens or hundreds of instances the scan cost is negligible. The real cost is operational: something must keep the connection counts accurate, which is what IncrementConnections and DecrementConnections do.

The code uses two layers of locking, and the reason is subtle. The balancer-level RWMutex protects the map itself from concurrent modification, while each InstanceMetrics has its own Mutex protecting its connection count. That split matters because counts change far more often than the set of instances changes. Locking the whole map for every connection increment would serialize all traffic on a single mutex. With per-instance locks, increments for different instances proceed in parallel, which is exactly the behavior you want under high concurrency. The DecrementConnections guard against negative counts also prevents drift from double-decrement bugs.

package main

import (
	"context"
	"fmt"
	"sync"
)

// InstanceMetrics tracks metrics for an instance
type InstanceMetrics struct {
	Address     string
	Connections int
	mu          sync.Mutex
}

// LeastConnectionsBalancer implements least connections load balancing
type LeastConnectionsBalancer struct {
	instances map[string]*InstanceMetrics
	mu        sync.RWMutex
}

// NewLeastConnectionsBalancer creates a new least connections balancer
func NewLeastConnectionsBalancer(addresses []string) *LeastConnectionsBalancer {
	instances := make(map[string]*InstanceMetrics)
	for _, addr := range addresses {
		instances[addr] = &InstanceMetrics{Address: addr}
	}
	return &LeastConnectionsBalancer{
		instances: instances,
	}
}

// SelectInstance selects the instance with least connections
func (b *LeastConnectionsBalancer) SelectInstance(ctx context.Context) (string, error) {
	b.mu.RLock()
	defer b.mu.RUnlock()

	if len(b.instances) == 0 {
		return "", fmt.Errorf("no instances available")
	}

	var selected *InstanceMetrics
	for _, instance := range b.instances {
		if selected == nil || instance.Connections < selected.Connections {
			selected = instance
		}
	}

	return selected.Address, nil
}

// IncrementConnections increments connection count
func (b *LeastConnectionsBalancer) IncrementConnections(address string) {
	b.mu.RLock()
	instance, exists := b.instances[address]
	b.mu.RUnlock()

	if exists {
		instance.mu.Lock()
		instance.Connections++
		instance.mu.Unlock()
	}
}

// DecrementConnections decrements connection count
func (b *LeastConnectionsBalancer) DecrementConnections(address string) {
	b.mu.RLock()
	instance, exists := b.instances[address]
	b.mu.RUnlock()

	if exists {
		instance.mu.Lock()
		if instance.Connections > 0 {
			instance.Connections--
		}
		instance.mu.Unlock()
	}
}

Least-connections is only as good as the connection accounting around it. The balancer itself never calls the increment and decrement methods, so the HTTP client layer must invoke them at request start and finish. Leaked increments happen when a request never reaches the decrement path, which is why production wrappers pair the decrement with defer. That same discipline applies to any stateful balancing strategy. Use least-connections when request processing times vary meaningfully across instances, so that active-connection counts are a better proxy for load than request counts.

Weighted Load Balancer

Weighted balancing lets you express capacity differences explicitly. Instead of treating every instance equally, you assign each one a weight proportional to how much traffic it should absorb. The WeightedBalancer below stores a slice of WeightedInstance values and precomputes the total weight in the constructor. Selection works by picking a random number between zero and the total weight, then walking the slice and subtracting each instance’s weight until the running total crosses the random value. An instance with weight 6 is six times more likely to be chosen than an instance with weight 1, which is exactly the ratio you want for a six-times-larger instance.

There are two implementation details worth noticing. First, the randomness lives in the balancer rather than in the caller, so concurrent callers do not coordinate on a shared generator; math/rand is safe for concurrent use in modern Go. Second, the method handles the degenerate cases explicitly — an empty list and a zero total weight both return descriptive errors instead of panicking or looping forever. The main limitation of static weights is that they are guesses. They must be adjusted as instances are resized or their actual performance drifts, which is why the earlier UpdateInstances pattern matters here too: weights should be refreshed from the same source that drives discovery.

package main

import (
	"context"
	"fmt"
	"math/rand"
	"sync"
)

// WeightedInstance represents an instance with weight
type WeightedInstance struct {
	Address string
	Weight  int
}

// WeightedBalancer implements weighted load balancing
type WeightedBalancer struct {
	instances []WeightedInstance
	totalWeight int
	mu        sync.RWMutex
}

// NewWeightedBalancer creates a new weighted balancer
func NewWeightedBalancer(instances []WeightedInstance) *WeightedBalancer {
	totalWeight := 0
	for _, instance := range instances {
		totalWeight += instance.Weight
	}
	return &WeightedBalancer{
		instances:   instances,
		totalWeight: totalWeight,
	}
}

// SelectInstance selects an instance based on weights
func (b *WeightedBalancer) SelectInstance(ctx context.Context) (string, error) {
	b.mu.RLock()
	defer b.mu.RUnlock()

	if len(b.instances) == 0 {
		return "", fmt.Errorf("no instances available")
	}

	if b.totalWeight == 0 {
		return "", fmt.Errorf("total weight is zero")
	}

	random := rand.Intn(b.totalWeight)
	cumulative := 0

	for _, instance := range b.instances {
		cumulative += instance.Weight
		if random < cumulative {
			return instance.Address, nil
		}
	}

	return b.instances[len(b.instances)-1].Address, nil
}

The weighted strategy is the first one that lets you express an opinion about capacity. Round-robin assumes equal instances, least-connections observes live load, and weighted balancing encodes expected capacity directly. In practice, weights are usually derived from instance size or measured throughput and refreshed through the discovery watch channel. The three strategies are not mutually exclusive — a weighted selection with least-connections fallback is a common production combination.

Health Checking

Discovery tells you which instances are supposed to exist, but it does not tell you which ones are currently able to serve traffic. A crashed process might still be registered; a network partition might make a live instance unreachable from your vantage point. Health checking closes that gap by actively probing instances and removing unhealthy ones from the candidate pool. The HealthChecker below performs periodic HTTP checks against each instance’s /health endpoint and records the results in a map guarded by a read-write mutex. Balancers then consult this map before selecting an instance, so requests never reach a dead backend.

Three design decisions in the code deserve attention. First, the HTTP client has an explicit two-second timeout. A health probe that hangs defeats the purpose of health checking, so the timeout ensures a dead instance is marked unhealthy within a bounded time. Second, CheckHealth takes a context, which lets the surrounding cancellation signal abort in-flight probes during shutdown. Third, the status map uses an RWMutex because reads, via GetHealthyInstances, are far more frequent than the writes performed by the ticker loop. The interval passed to StartHealthChecking is the key tuning parameter, and the trade-off is direct: shorter intervals catch failures faster but generate more probe traffic.

package main

import (
	"context"
	"fmt"
	"net/http"
	"sync"
	"time"
)

// HealthChecker checks service health
type HealthChecker struct {
	instances map[string]bool
	mu        sync.RWMutex
	client    *http.Client
}

// NewHealthChecker creates a new health checker
func NewHealthChecker() *HealthChecker {
	return &HealthChecker{
		instances: make(map[string]bool),
		client: &http.Client{
			Timeout: 2 * time.Second,
		},
	}
}

// CheckHealth checks if an instance is healthy
func (hc *HealthChecker) CheckHealth(ctx context.Context, address string) bool {
	url := fmt.Sprintf("http://%s/health", address)

	req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
	if err != nil {
		return false
	}

	resp, err := hc.client.Do(req)
	if err != nil {
		return false
	}
	defer resp.Body.Close()

	return resp.StatusCode == http.StatusOK
}

// StartHealthChecking starts periodic health checks
func (hc *HealthChecker) StartHealthChecking(ctx context.Context, instances []string, interval time.Duration) {
	ticker := time.NewTicker(interval)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			for _, instance := range instances {
				healthy := hc.CheckHealth(ctx, instance)
				hc.mu.Lock()
				hc.instances[instance] = healthy
				hc.mu.Unlock()
			}
		}
	}
}

// GetHealthyInstances returns only healthy instances
func (hc *HealthChecker) GetHealthyInstances(instances []string) []string {
	hc.mu.RLock()
	defer hc.mu.RUnlock()

	var healthy []string
	for _, instance := range instances {
		if isHealthy, exists := hc.instances[instance]; exists && isHealthy {
			healthy = append(healthy, instance)
		}
	}
	return healthy
}

Health checking is the glue that makes discovery and balancing trustworthy. Without it, a balancer will cheerfully send traffic to a dead instance and depend on timeouts and retries to mask the failure. Note that this implementation treats an instance as healthy only if it was probed recently; an instance missing from the map is filtered out, which is the fail-closed behavior you want. For production, prefer active probing of a lightweight /health endpoint over passive TCP checks, and add a readiness endpoint that reflects the instance’s true ability to serve work.

Client-Side Load Balancing

Everything so far — discovery, balancing strategies, and health checking — comes together in the client-side load balancing pattern. The idea is that the calling service itself performs discovery and selection instead of sending requests through a shared proxy or load balancer. This eliminates a network hop and a single point of failure, at the cost of pushing the balancing logic into every client. The ClientSideLoadBalancer below orchestrates the three concerns behind two small interfaces. Discovery wraps any discovery mechanism, and LoadBalancer wraps any selection strategy. That interface design is deliberate: it lets you swap DNS for Consul, or round-robin for weighted, without touching the orchestration code.

The DoRequest method shows the correct order of operations. First it discovers instances, then it filters them through the health checker, then it feeds the healthy set to the balancer, and finally it selects one and issues the HTTP request. Each step narrows the candidate pool: from all registered instances to healthy ones to a single chosen instance. The request carries a context, so cancellation propagates from the caller through the HTTP call. The two key interfaces make this component testable — you can inject fake discovery and balancer implementations to test the orchestration in isolation, without any real network calls.

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"time"
)

// ClientSideLoadBalancer implements client-side load balancing
type ClientSideLoadBalancer struct {
	discovery Discovery
	balancer  LoadBalancer
	healthChecker *HealthChecker
	client    *http.Client
}

// Discovery interface for service discovery
type Discovery interface {
	DiscoverService(ctx context.Context, serviceName string) ([]string, error)
}

// LoadBalancer interface for load balancing
type LoadBalancer interface {
	SelectInstance(ctx context.Context) (string, error)
	UpdateInstances(instances []string)
}

// NewClientSideLoadBalancer creates a new client-side load balancer
func NewClientSideLoadBalancer(discovery Discovery, balancer LoadBalancer) *ClientSideLoadBalancer {
	return &ClientSideLoadBalancer{
		discovery: discovery,
		balancer:  balancer,
		healthChecker: NewHealthChecker(),
		client: &http.Client{
			Timeout: 5 * time.Second,
		},
	}
}

// DoRequest performs a request with load balancing
func (lb *ClientSideLoadBalancer) DoRequest(ctx context.Context, serviceName, path string) ([]byte, error) {
	// Discover service instances
	instances, err := lb.discovery.DiscoverService(ctx, serviceName)
	if err != nil {
		return nil, err
	}

	// Filter healthy instances
	healthyInstances := lb.healthChecker.GetHealthyInstances(instances)
	if len(healthyInstances) == 0 {
		return nil, fmt.Errorf("no healthy instances available")
	}

	lb.balancer.UpdateInstances(healthyInstances)

	// Select instance
	instance, err := lb.balancer.SelectInstance(ctx)
	if err != nil {
		return nil, err
	}

	// Make request
	url := fmt.Sprintf("http://%s%s", instance, path)
	req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
	if err != nil {
		return nil, err
	}

	resp, err := lb.client.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()

	return io.ReadAll(resp.Body)
}

Client-side load balancing is the pattern used by most service meshes and by many distributed systems libraries. Its biggest operational advantage is that it removes the load balancer as a network hop and as a single point of failure. Its biggest cost is that every language runtime must implement discovery and selection, and every client must be updated when the logic changes. Choose client-side balancing when you control all the clients and want maximum resilience. Choose a server-side proxy or ingress when you need centralized policy, observability, or support for clients you do not control.

Best Practices

The components built in this article work together, but the difference between a demo and a production system is in the details around them. The three practices below — health check tuning, retry logic, and circuit breaking — address the most common causes of failure in real deployments. They are small additions to the core loop that dramatically change its behavior under load. Each one also has its own tuning parameters and trade-offs, so treat them as a starting point rather than a finished configuration.

1. Health Check Configuration

The health check interval is the first lever to tune. A ten-second interval shown below is a reasonable default: it catches most failures within a few seconds without hammering the fleet with probe traffic. The right value depends on how quickly your services can actually fail and how fast your monitoring needs to react. Longer-lived batch workloads can tolerate longer intervals; user-facing request paths benefit from faster detection. Whatever interval you choose, remember that the probe timeout must stay comfortably below the interval, or probes will stack up and overwhelm the instance being checked.

// Configure appropriate health check intervals
healthChecker.StartHealthChecking(ctx, instances, 10*time.Second)

The configuration shown is deliberately small, but the decision behind it is not. Health check intervals are a classic availability-versus-overhead trade-off, and getting them wrong manifests as either slow failover or a probe-induced denial of service. Start with ten seconds, measure your real failure detection time, and adjust based on data rather than guesswork.

2. Retry Logic

Transient failures are inevitable in a distributed system. A connection resets, a request times out, or an instance dies between discovery and selection. Retrying the request against the same or a different instance absorbs most of these failures without any user-visible impact. The DoRequestWithRetry wrapper below retries up to maxRetries times, calling the same DoRequest each time so each attempt goes through discovery and balancing anew. That is the crucial property: because DoRequest re-selects an instance on every call, a retry naturally falls over to a healthy instance when the first choice fails.

The backoff in the code grows with the attempt number — 100ms, then 200ms, then 300ms. That increasing pause is deliberate, because hammering a failing service with immediate retries often makes the problem worse. The design also retains the last error and returns it if all attempts fail, so the caller can distinguish a transient failure that recovered from a persistent outage. The main pitfall is retrying requests that are not idempotent, since a retried write can be applied twice. For non-idempotent operations, retry only after you can prove the request was not processed, or route retries through a client that deduplicates.

func (lb *ClientSideLoadBalancer) DoRequestWithRetry(ctx context.Context, serviceName, path string, maxRetries int) ([]byte, error) {
	var lastErr error
	for attempt := 0; attempt < maxRetries; attempt++ {
		data, err := lb.DoRequest(ctx, serviceName, path)
		if err == nil {
			return data, nil
		}
		lastErr = err
		time.Sleep(time.Duration(attempt+1) * 100 * time.Millisecond)
	}
	return nil, lastErr
}

The pattern generalizes beyond HTTP to any remote call, and it composes with the health checker. A retry after a timeout is an opportunity for the health checker to have marked the dead instance unhealthy. Keep the retry count small — two or three attempts is typical — and always back off, because unbounded retries under load become a thundering herd.

3. Circuit Breaker Pattern

Retries help with transient failures, but they fail badly when a service is down for an extended period. Every retry wastes resources on a dependency that is definitely broken, and the wasted work can cascade into a full outage. The circuit breaker prevents that by failing fast once the failure count exceeds a threshold. The struct below holds the state machine: closed (normal operation), open (fail fast), and half-open (probing whether the dependency recovered). failureThreshold controls when the circuit opens, successThreshold controls when it closes again, and timeout controls how long the circuit stays open before probing.

type CircuitBreaker struct {
	failureThreshold int
	successThreshold int
	timeout          time.Duration
	state            string // "closed", "open", "half-open"
	failures         int
	successes        int
	lastFailureTime  time.Time
}

The three states encode the recovery dance. In closed, calls pass through and failures are counted. When failures reach the threshold, the circuit flips to open and all calls fail fast without touching the dependency. After timeout elapses, it moves to half-open, letting a small number of probe requests through. If those succeed, the circuit closes; if any fail, it opens again. The tuning variables are the thresholds and timeout, and getting them right requires knowing your service’s typical failure recovery time. Circuit breakers belong on every call to a downstream service, and they work especially well alongside the retry logic above.

Common Pitfalls

The patterns above solve the main problems, but production systems still fail in predictable ways. The four pitfalls below are the ones this author has seen most often in real deployments. Each one is a small mistake with a large operational impact, and each has a straightforward fix.

1. Stale Instance Lists

Always refresh instance lists periodically. An instance list that is fetched once at startup and cached forever will keep routing traffic to retired nodes. Wire the discovery watch channel from earlier sections into the balancer’s UpdateInstances so the list stays current. Treat stale lists as the number-one source of mysterious connection failures in microservices.

2. Ignoring Health Status

Don’t send requests to unhealthy instances. If the balancer selects from the raw discovery result instead of the health-filtered set, it will send traffic to dead backends and rely on retries to paper over the problem. Always interpose GetHealthyInstances between discovery and selection, as the client-side balancer does.

3. No Retry Logic

Implement retry logic for transient failures. A single request attempt converts a blip in one instance into a user-visible error. A small retry loop with backoff, as shown in the best practices section, absorbs the blip entirely.

4. Unbalanced Load Distribution

Monitor and adjust weights based on actual performance. Static weights drift out of date as instances are resized or their load changes. Watch utilization metrics and refresh weights through the same update path used for instance lists.

Resources

• Service Discovery Patterns
• Load Balancing Algorithms
• Consul Service Discovery
• Go HTTP Client

Summary

Service discovery and load balancing are essential for microservices. Key takeaways:

• Use DNS or service registries for discovery
• Implement appropriate load balancing strategies
• Monitor service health continuously
• Implement retry logic and circuit breakers
• Use client-side or server-side balancing appropriately
• Keep instance lists fresh and accurate

These patterns ensure your microservices remain resilient and performant at scale.