Reinforcement Learning Algorithms: From Q-Learning to Deep Q-Networks

Introduction

Reinforcement Learning (RL) represents a fundamentally different paradigm from supervised and unsupervised learning. Instead of learning from labeled data or finding patterns in unlabeled data, RL agents learn through interaction with an environment, receiving rewards for their actions and learning to maximize cumulative reward over time. In 2026, RL has achieved remarkable successes, from beating world champions in complex games to robots that learn to walk and manipulate objects, to systems that optimize computer networks and financial portfolios.

The key insight of reinforcement learning is that intelligence emerges from the interaction between an agent and its environment. An agent takes actions, the environment responds with new states and rewards, and the agent learns from this feedback loop to improve its behavior. This learning paradigm mirrors how humans and animals learn through trial and error, making RL one of the most general and ambitious forms of machine learning.

The Reinforcement Learning Framework

Agent-Environment Interaction

The RL problem is formalized as a Markov Decision Process (MDP). At each time step, the agent observes the current state s_t from the environment, takes an action a_t, and receives a reward r_t and the next state s_{t+1}. This process continues until a terminal state is reached or for a fixed number of steps.

The agent’s behavior is defined by a policy π(a|s), which maps states to probability distributions over actions. The goal is to find a policy that maximizes the expected cumulative reward, often called the return. The return is typically discounted to prioritize immediate rewards over distant ones: G_t = r_t + γr_{t+1} + γ²r_{t+2} + …

Value Functions and Bellman Equations

Central to RL are value functions that estimate how good states or state-action pairs are. The state-value function V^π(s) represents the expected return when starting from state s and following policy π. The action-value function Q^π(s,a) represents the expected return when taking action a from state s and then following policy π.

The Bellman equations provide recursive relationships for value functions:

V^π(s) = Σ_a π(a|s) Σ_{s',r} p(s',r|s,a)[r + γV^π(s')]
Q^π(s,a) = Σ_{s',r} p(s',r|s,a)[r + γ Σ_{a'} π(a'|s') Q^π(s',a')]

These equations are the foundation for many RL algorithms, expressing how values propagate backward from rewards through the state space.

Tabular Learning Methods

Q-Learning

Q-Learning is one of the most fundamental RL algorithms, learning optimal action values through temporal-difference updates. For each state-action pair, Q-learning maintains an estimate Q(s,a) and updates it based on the observed reward and the best possible future value:

Q(s,a) ← Q(s,a) + α[r + γ max_{a'} Q(s',a') - Q(s,a)]

Where α is the learning rate and γ is the discount factor. This update gradually moves Q values toward the optimal action values, eventually learning the optimal policy regardless of the exploration strategy used.

Q-learning is guaranteed to converge to optimal values given sufficient exploration and appropriate learning rate decay. However, it struggles with large state spaces where representing Q for every state-action pair becomes impractical.

SARSA

SARSA (State-Action-Reward-State-Action) is an on-policy algorithm that learns Q-values for the current policy rather than the optimal policy:

Q(s,a) ← Q(s,a) + α[r + γ Q(s',a') - Q(s,a)]

Where a’ is the action actually taken in state s’. This makes SARSA more conservative than Q-learning, as it accounts for the exploration being performed. SARSA is particularly useful when exploration costs are high or when learning policies that account for exploration noise.

Deep Q-Networks (DQN)

From Tables to Neural Networks

When state spaces are large or continuous, tabular Q-learning becomes impractical. Deep Q-Networks (DQN) address this by using deep neural networks to approximate Q-values. The network takes states as input and outputs Q-values for all actions.

The key innovation of DQN is experience replay and target networks. Experience replay stores transitions (s, a, r, s’) in a replay buffer and samples batches for training, breaking temporal correlation in the data. The target network provides stable targets for the Bellman equation, updated periodically rather than at every step.

The DQN loss is:

L = E[(r + γ max_{a'} Q_target(s',a') - Q_online(s,a))²]

Improvements to DQN

Several improvements have enhanced DQN’s performance and stability. Double DQN addresses overestimation by using two networks to select and evaluate actions. Dueling DQN decomposes Q-values into state-value and advantage functions, allowing better estimation of state values independently of actions. Prioritized Experience Replay samples important transitions more frequently based on TD error.

Rainbow DQN combines multiple improvements: double DQN, prioritized replay, dueling networks, distributional RL, and noisy networks. This combination achieved state-of-the-art results on the Atari benchmark.

Policy Gradient Methods

REINFORCE

Policy gradient methods directly optimize the policy without explicitly estimating value functions. REINFORCE updates policy parameters in the direction of gradient ascent on expected return:

∇J(θ) = E[∇_θ log π_θ(a|s) · G_t]

Where π_θ is the parameterized policy. This gradient tells us how to adjust parameters to increase the probability of good actions. The algorithm is simple but has high variance; returns can vary widely across episodes, making learning unstable.

Actor-Critic Methods

Actor-critic methods combine policy gradients with value function approximation to reduce variance. The actor updates the policy parameters, while the critic estimates value functions to provide baselines. Common approaches include A2C/A3C (Advantage Actor-Critic) and PPO (Proximal Policy Optimization).

A2C uses an advantage function A(s,a) = Q(s,a) - V(s) instead of raw returns, reducing variance while maintaining unbiased gradient estimates. The advantage measures how much better an action is compared to the average.

Proximal Policy Optimization (PPO)

PPO has become one of the most popular RL algorithms due to its simplicity and effectiveness. It constrains policy updates to prevent destructive large changes:

L^CLIP(θ) = E[min(r_t(θ) Â_t, clip(r_t(θ), 1-ε, 1+ε) Â_t)]

Where r_t(θ) is the probability ratio between new and old policies, and ε is a small hyperparameter (typically 0.2). This clipped objective prevents the policy from changing too dramatically while still allowing learning.

PPO has succeeded in continuous control tasks, robotics, and has been foundational in large-scale RL applications like OpenAI’s ChatGPT training.

Advanced RL Topics

Multi-Agent Reinforcement Learning

When multiple agents interact, the environment becomes non-stationary from any single agent’s perspective. Multi-agent RL (MARL) introduces additional challenges: coordination, competition, and emergent behaviors. Topics include cooperative MARL, competitive MARL (like in game-playing), and mixed settings.

Hierarchical RL

Hierarchical RL decomposes complex tasks into subtasks, learning at multiple temporal scales. Options framework defines temporally extended actions (options) that can be composed. This approach enables faster learning and transfer across tasks.

Meta-Learning in RL

Meta-learning for RL focuses on learning to learn—acquiring knowledge that can quickly adapt to new tasks. Model-Agnostic Meta-Learning (MAML) learns initial parameters that can be quickly fine-tuned with few gradient steps on new tasks. This is crucial for sample-efficient RL in real-world applications.

Offline Reinforcement Learning

Offline (or batch) RL learns from fixed datasets without interaction. This is practical for applications where online exploration is dangerous or expensive. Conservative Q-Learning (CQL) and decision transformers have shown promise in offline settings, learning effective policies from static datasets.

Applications of Reinforcement Learning

Game Playing

RL has achieved superhuman performance in complex games. AlphaGo combined tree search with neural networks to beat world champions in Go. OpenAI Five learned to play Dota 2 through self-play. These systems demonstrate RL’s ability to handle enormous state and action spaces through clever combination of learning and search.

Robotics

RL enables robots to learn complex manipulation skills, locomotion, and navigation without explicit programming. Robots can learn to grasp novel objects, perform surgical maneuvers, and navigate complex environments. Sim-to-real transfer (training in simulation, deploying in reality) has made robotic RL practical.

Resource Management

RL optimizes computing resources, including job scheduling in data centers, network routing, and power grid management. These systems must make sequential decisions under uncertainty, a natural fit for RL.

Finance

In algorithmic trading and portfolio management, RL learns trading strategies that adapt to market conditions. While challenging due to non-stationarity and risk considerations, RL offers a data-driven approach to financial optimization.

Implementing RL Algorithms

Basic Q-Learning Implementation

A simple tabular Q-learning implementation:

import numpy as np

class QLearningAgent:
    def __init__(self, n_states, n_actions, alpha=0.1, gamma=0.95, epsilon=0.1):
        self.q_table = np.zeros((n_states, n_actions))
        self.alpha = alpha  # learning rate
        self.gamma = gamma  # discount factor
        self.epsilon = epsilon  # exploration rate
    
    def choose_action(self, state):
        if np.random.random() < self.epsilon:
            return np.random.randint(len(self.q_table[state]))
        return np.argmax(self.q_table[state])
    
    def learn(self, state, action, reward, next_state):
        best_next = np.max(self.q_table[next_state])
        td_target = reward + self.gamma * best_next
        td_error = td_target - self.q_table[state][action]
        self.q_table[state][action] += self.alpha * td_error

DQN with PyTorch

A basic DQN implementation uses a neural network to approximate Q-values:

import torch
import torch.nn as nn

class DQN(nn.Module):
    def __init__(self, state_dim, action_dim):
        super(DQN, self).__init__()
        self.network = nn.Sequential(
            nn.Linear(state_dim, 128),
            nn.ReLU(),
            nn.Linear(128, 128),
            nn.ReLU(),
            nn.Linear(128, action_dim)
        )
    
    def forward(self, x):
        return self.network(x)

Resources

• Sutton & Barto - Reinforcement Learning: An Introduction
• OpenAI Spinning Up
• Stable Baselines3
• DeepMind’s RL Course

Conclusion

Reinforcement learning has matured into a powerful paradigm for sequential decision-making. From fundamental algorithms like Q-learning to modern methods like PPO and offline RL, the field offers tools for solving problems where optimal behavior must be learned through interaction. As RL algorithms become more sample-efficient and reliable, their adoption in robotics, resource management, and AI systems will continue to grow.